[jira] [Comment Edited] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878946#comment-16878946
]
Rene Cordier edited comment on JAMES-2806 at 7/5/19 3:23 AM:
-
Work for the ObjectStorage implementation has been divided into 3 PRs :
* [https://github.com/linagora/james-project/pull/2489] `[Part 1]
ObjectStorageBlobsDAO with BucketName`
* [https://github.com/linagora/james-project/pull/2492] `[Part 2]
ObjectStorageBlobsDAO creating bucket when saving fails (Swfit + S3 )`
* [https://github.com/linagora/james-project/pull/2497] `[Part 3]
ObjectStorageBlobsDAO BucketPrefix implementation`
was (Author: rcordier):
Work for the ObjectStorage implementation has been divided into 3 PRs :
* [https://github.com/linagora/james-project/pull/2489] `[Part 1]
ObjectStorageBlobsDAO with BucketName`
* [https://github.com/linagora/james-project/pull/2492] `[Part 2]
ObjectStorageBlobsDAO creating bucket when saving fails (Swfit + S3 )`
* [https://github.com/linagora/james-project/pull/2497] `[Part 3]
ObjectStorageBlobsDAO BucketPrefix implementation`
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Comment Edited] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878946#comment-16878946
]
Rene Cordier edited comment on JAMES-2806 at 7/5/19 3:22 AM:
-
Work for the ObjectStorage implementation has been divided into 3 PRs :
* [https://github.com/linagora/james-project/pull/2489] `[Part 1]
ObjectStorageBlobsDAO with BucketName`
* [https://github.com/linagora/james-project/pull/2492] `[Part 2]
ObjectStorageBlobsDAO creating bucket when saving fails (Swfit + S3 )`
* [https://github.com/linagora/james-project/pull/2497] `[Part 3]
ObjectStorageBlobsDAO BucketPrefix implementation`
was (Author: rcordier):
Work for the ObjectStorage implementation has been divided into 3 PRs :
* https://github.com/linagora/james-project/pull/2489 `[JAMES-2806] [Part 1]
ObjectStorageBlobsDAO with BucketName`
* https://github.com/linagora/james-project/pull/2492 `[JAMES-2806] [Part 2]
ObjectStorageBlobsDAO creating bucket when saving fails (Swfit + S3 )`
* https://github.com/linagora/james-project/pull/2497 `[JAMES-2806] [Part 3]
ObjectStorageBlobsDAO BucketPrefix implementation`
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878946#comment-16878946
]
Rene Cordier commented on JAMES-2806:
-
Work for the ObjectStorage implementation has been divided into 3 PRs :
* https://github.com/linagora/james-project/pull/2489 `[JAMES-2806] [Part 1]
ObjectStorageBlobsDAO with BucketName`
* https://github.com/linagora/james-project/pull/2492 `[JAMES-2806] [Part 2]
ObjectStorageBlobsDAO creating bucket when saving fails (Swfit + S3 )`
* https://github.com/linagora/james-project/pull/2497 `[JAMES-2806] [Part 3]
ObjectStorageBlobsDAO BucketPrefix implementation`
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2773) Add glowroot in the docker images
[ https://issues.apache.org/jira/browse/JAMES-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat closed JAMES-2773. - > Add glowroot in the docker images > - > > Key: JAMES-2773 > URL: https://issues.apache.org/jira/browse/JAMES-2773 > Project: James Server > Issue Type: Bug >Reporter: Rémi Kowalski >Priority: Major > Fix For: 3.4.0 > > > In order to have instrumentation on the JMAP and IMAP transactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Reopened] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Duprat reopened JAMES-2806:
---
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2816) Append messages optimization
[ https://issues.apache.org/jira/browse/JAMES-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat resolved JAMES-2816. --- Resolution: Fixed Fix Version/s: 3.4.0 merged > Append messages optimization > > > Key: JAMES-2816 > URL: https://issues.apache.org/jira/browse/JAMES-2816 > Project: James Server > Issue Type: Improvement >Reporter: Gautier DI FOLCO >Priority: Trivial > Fix For: 3.4.0 > > > We tend to copy the whole message when only meta-data are needed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2817) Various EventBus fixes
[ https://issues.apache.org/jira/browse/JAMES-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat resolved JAMES-2817. --- Resolution: Fixed Fix Version/s: 3.4.0 merged > Various EventBus fixes > -- > > Key: JAMES-2817 > URL: https://issues.apache.org/jira/browse/JAMES-2817 > Project: James Server > Issue Type: Improvement >Reporter: Gautier DI FOLCO >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2817) Various EventBus fixes
[ https://issues.apache.org/jira/browse/JAMES-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat closed JAMES-2817. - > Various EventBus fixes > -- > > Key: JAMES-2817 > URL: https://issues.apache.org/jira/browse/JAMES-2817 > Project: James Server > Issue Type: Improvement >Reporter: Gautier DI FOLCO >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2816) Append messages optimization
[ https://issues.apache.org/jira/browse/JAMES-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat closed JAMES-2816. - > Append messages optimization > > > Key: JAMES-2816 > URL: https://issues.apache.org/jira/browse/JAMES-2816 > Project: James Server > Issue Type: Improvement >Reporter: Gautier DI FOLCO >Priority: Trivial > Fix For: 3.4.0 > > > We tend to copy the whole message when only meta-data are needed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 09/15: JAMES-2817 Rename subscribeWorkQueue to consumeWorkQueue and subscribe in start()
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 87136c39e011e6d0197e3a0e85a8203c1e861e35
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 09:14:26 2019 +0100
JAMES-2817 Rename subscribeWorkQueue to consumeWorkQueue and subscribe in
start()
---
.../apache/james/mailbox/events/GroupRegistration.java | 18 +-
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
index bbdd46b..7730efc 100644
---
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
+++
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
@@ -37,7 +37,6 @@ import org.apache.james.util.MDCBuilder;
import com.github.fge.lambdas.Throwing;
import com.google.common.base.Preconditions;
import com.rabbitmq.client.Connection;
-
import reactor.core.Disposable;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
@@ -105,10 +104,11 @@ class GroupRegistration implements Registration {
}
GroupRegistration start() {
-createGroupWorkQueue()
-.then(retryHandler.createRetryExchange(queueName))
-.doOnSuccess(any -> this.subscribeWorkQueue())
-.block();
+receiverSubscriber = Optional
+.of(createGroupWorkQueue()
+.then(retryHandler.createRetryExchange(queueName))
+.then(Mono.fromCallable(() -> this.consumeWorkQueue()))
+.block());
return this;
}
@@ -126,12 +126,12 @@ class GroupRegistration implements Registration {
.then();
}
-private void subscribeWorkQueue() {
-receiverSubscriber =
Optional.of(receiver.consumeManualAck(queueName.asString(), new
ConsumeOptions().qos(EventBus.EXECUTION_RATE))
+private Disposable consumeWorkQueue() {
+return receiver.consumeManualAck(queueName.asString(), new
ConsumeOptions().qos(EventBus.EXECUTION_RATE))
+.publishOn(Schedulers.parallel())
.filter(delivery -> Objects.nonNull(delivery.getBody()))
.flatMap(this::deliver)
-.subscribeOn(Schedulers.elastic())
-.subscribe());
+.subscribe();
}
private Mono deliver(AcknowledgableDelivery acknowledgableDelivery) {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 03/15: Allow Store.Impl to cary along a byte[] representation
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 30dc0b9f04d5bd3fc7121aedf8af8a972a356d35
Author: Benoit Tellier
AuthorDate: Fri Dec 7 15:08:00 2018 +0700
Allow Store.Impl to cary along a byte[] representation
Bytes are available but hidded behing an inputStream for implementation
purposes.
We propose in this commit a new level of indirection allowing to save an
object as bytes or as InputSteam.
---
.../main/java/org/apache/james/blob/api/Store.java | 24 ++
.../apache/james/blob/mail/MimeMessageStore.java | 6 +++---
2 files changed, 23 insertions(+), 7 deletions(-)
diff --git
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/Store.java
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/Store.java
index 6a4f859..ce640d5 100644
--- a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/Store.java
+++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/Store.java
@@ -19,7 +19,6 @@
package org.apache.james.blob.api;
-import java.io.InputStream;
import java.util.Collection;
import java.util.Objects;
import java.util.stream.Stream;
@@ -66,8 +65,25 @@ public interface Store {
class Impl implements Store {
+public interface ValueToSave {
+Mono saveIn(BucketName bucketName, BlobStore blobStore);
+}
+
+public static class BytesToSave implements ValueToSave {
+private final byte[] bytes;
+
+public BytesToSave(byte[] bytes) {
+this.bytes = bytes;
+}
+
+@Override
+public Mono saveIn(BucketName bucketName, BlobStore
blobStore) {
+return blobStore.save(bucketName, bytes);
+}
+}
+
public interface Encoder {
-Stream> encode(T t);
+Stream> encode(T t);
}
public interface Decoder {
@@ -94,9 +110,9 @@ public interface Store {
.map(idFactory::generate);
}
-private Mono> saveEntry(Pair entry) {
+private Mono> saveEntry(Pair entry) {
return Mono.just(entry.getLeft())
-.zipWith(blobStore.save(BucketName.DEFAULT, entry.getRight()));
+.zipWith(entry.getRight().saveIn(BucketName.DEFAULT,
blobStore));
}
@Override
diff --git
a/server/blob/mail-store/src/main/java/org/apache/james/blob/mail/MimeMessageStore.java
b/server/blob/mail-store/src/main/java/org/apache/james/blob/mail/MimeMessageStore.java
index 12683f9..f0c2480 100644
---
a/server/blob/mail-store/src/main/java/org/apache/james/blob/mail/MimeMessageStore.java
+++
b/server/blob/mail-store/src/main/java/org/apache/james/blob/mail/MimeMessageStore.java
@@ -69,15 +69,15 @@ public class MimeMessageStore {
static class MimeMessageEncoder implements Store.Impl.Encoder
{
@Override
-public Stream> encode(MimeMessage message)
{
+public Stream>
encode(MimeMessage message) {
try {
byte[] messageAsArray = messageToArray(message);
int bodyStartOctet = computeBodyStartOctet(messageAsArray);
byte[] headerBytes = getHeaderBytes(messageAsArray,
bodyStartOctet);
byte[] bodyBytes = getBodyBytes(messageAsArray,
bodyStartOctet);
return Stream.of(
-Pair.of(HEADER_BLOB_TYPE, new
ByteArrayInputStream(headerBytes)),
-Pair.of(BODY_BLOB_TYPE, new
ByteArrayInputStream(bodyBytes)));
+Pair.of(HEADER_BLOB_TYPE, new
Store.Impl.BytesToSave(headerBytes)),
+Pair.of(BODY_BLOB_TYPE, new
Store.Impl.BytesToSave(bodyBytes)));
} catch (MessagingException | IOException e) {
throw new RuntimeException(e);
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 10/15: JAMES-2817 Avoid C-like cast syntax in GroupRegistration
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 37b0b39571f9f50268bdfa88979ace839bd84369
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 09:47:20 2019 +0100
JAMES-2817 Avoid C-like cast syntax in GroupRegistration
---
.../main/java/org/apache/james/mailbox/events/GroupRegistration.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
index 7730efc..c704ad3 100644
---
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
+++
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
@@ -164,7 +164,7 @@ class GroupRegistration implements Registration {
return
Optional.ofNullable(acknowledgableDelivery.getProperties().getHeaders())
.flatMap(headers -> Optional.ofNullable(headers.get(RETRY_COUNT)))
.filter(object -> object instanceof Integer)
-.map(object -> (Integer) object)
+.map(Integer.class::cast)
.orElse(DEFAULT_RETRY_COUNT);
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 15/15: Merge remote-tracking branch 'mbaechler/some-small-eventBus-fix'
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit 2548451538098db2912872d06c0421b38be5f372 Merge: 031775a 2338e64 Author: Antoine DUPRAT AuthorDate: Thu Jul 4 11:29:38 2019 +0200 Merge remote-tracking branch 'mbaechler/some-small-eventBus-fix' .../mailbox/events/ErrorHandlingContract.java | 4 ++-- .../james/mailbox/events/GroupRegistration.java| 25 ++ .../james/mailbox/events/WaitDelayGenerator.java | 24 - .../mailbox/events/WaitDelayGeneratorTest.java | 8 +++ 4 files changed, 31 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 08/15: JAMES-2816 Refactor appendMessage method to make it more readable
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 02fb91af65229e1ccd4f714c55682b7abe4673f5
Author: Rémi Kowalski
AuthorDate: Mon Jul 1 17:44:22 2019 +0200
JAMES-2816 Refactor appendMessage method to make it more readable
---
.../james/mailbox/store/StoreMessageManager.java | 276 -
1 file changed, 160 insertions(+), 116 deletions(-)
diff --git
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
index 5015d67..3cf9e06 100644
---
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
+++
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
@@ -133,6 +133,16 @@ public class StoreMessageManager implements MessageManager
{
*/
protected static final Flags MINIMAL_PERMANET_FLAGS;
+private static class MediaType {
+final String mediaType;
+final String subType;
+
+private MediaType(String mediaType, String subType) {
+this.mediaType = mediaType;
+this.subType = subType;
+}
+}
+
static {
MINIMAL_PERMANET_FLAGS = new Flags();
MINIMAL_PERMANET_FLAGS.add(Flags.Flag.ANSWERED);
@@ -327,7 +337,6 @@ public class StoreMessageManager implements MessageManager {
@Override
public ComposedMessageId appendMessage(InputStream msgIn, Date
internalDate, final MailboxSession mailboxSession, boolean isRecent, Flags
flagsToBeSet) throws MailboxException {
-
File file = null;
if (!isWriteable(mailboxSession)) {
@@ -346,126 +355,20 @@ public class StoreMessageManager implements
MessageManager {
// Disable line length... This should be handled by the smtp
server
// component and not the parser itself
// https://issues.apache.org/jira/browse/IMAP-122
-
-final MimeTokenStream parser = new
MimeTokenStream(MimeConfig.PERMISSIVE, new DefaultBodyDescriptorBuilder());
-
-parser.setRecursionMode(RecursionMode.M_NO_RECURSE);
-parser.parse(bIn);
-final HeaderImpl header = new HeaderImpl();
-
-EntityState next = parser.next();
-while (next != EntityState.T_BODY && next !=
EntityState.T_END_OF_STREAM && next != EntityState.T_START_MULTIPART) {
-if (next == EntityState.T_FIELD) {
-header.addField(parser.getField());
-}
-next = parser.next();
-}
+final MimeTokenStream parser = getParser(bIn);
+final HeaderImpl header = readHeader(parser);
final MaximalBodyDescriptor descriptor =
(MaximalBodyDescriptor) parser.getBodyDescriptor();
-final PropertyBuilder propertyBuilder = new PropertyBuilder();
-final String mediaType;
-final String mediaTypeFromHeader = descriptor.getMediaType();
-final String subType;
-if (mediaTypeFromHeader == null) {
-mediaType = "text";
-subType = "plain";
-} else {
-mediaType = mediaTypeFromHeader;
-subType = descriptor.getSubType();
-}
-propertyBuilder.setMediaType(mediaType);
-propertyBuilder.setSubType(subType);
-propertyBuilder.setContentID(descriptor.getContentId());
-
propertyBuilder.setContentDescription(descriptor.getContentDescription());
-
propertyBuilder.setContentLocation(descriptor.getContentLocation());
-propertyBuilder.setContentMD5(descriptor.getContentMD5Raw());
-
propertyBuilder.setContentTransferEncoding(descriptor.getTransferEncoding());
-
propertyBuilder.setContentLanguage(descriptor.getContentLanguage());
-
propertyBuilder.setContentDispositionType(descriptor.getContentDispositionType());
-
propertyBuilder.setContentDispositionParameters(descriptor.getContentDispositionParameters());
-
propertyBuilder.setContentTypeParameters(descriptor.getContentTypeParameters());
-// Add missing types
-final String codeset = descriptor.getCharset();
-if (codeset == null) {
-if ("TEXT".equalsIgnoreCase(mediaType)) {
-propertyBuilder.setCharset("us-ascii");
-}
-} else {
-propertyBuilder.setCharset(codeset);
-}
-
-final String boundary = descriptor.getBoundary();
-if (boundary != null) {
-
[james-project] 06/15: JAMES-2816 Use buffered streams to handle mails
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 55257ba20f52d0d91043e2f6733d77209520e2d7
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 15:16:34 2019 +0100
JAMES-2816 Use buffered streams to handle mails
x3 improvement in micro-benchmark, should be around 5% improvement in
real life
appendMessage reported to take 8.51% in a flamegraph
---
.../java/org/apache/james/mailbox/store/StoreMessageManager.java | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
index 8c32c55..5015d67 100644
---
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
+++
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
@@ -22,6 +22,8 @@ package org.apache.james.mailbox.store;
import static
org.apache.james.mailbox.extension.PreDeletionHook.DeleteOperation;
import static
org.apache.james.mailbox.store.mail.AbstractMessageMapper.UNLIMITED;
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
@@ -102,7 +104,6 @@ import com.github.steveash.guavate.Guavate;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSortedMap;
-
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
@@ -339,7 +340,8 @@ public class StoreMessageManager implements MessageManager {
// source for the InputStream
file = File.createTempFile("imap", ".msg");
try (FileOutputStream out = new FileOutputStream(file);
- TeeInputStream tmpMsgIn = new TeeInputStream(msgIn, out);
+ BufferedOutputStream bufferedOut = new
BufferedOutputStream(out);
+ BufferedInputStream tmpMsgIn = new BufferedInputStream(new
TeeInputStream(msgIn, bufferedOut));
BodyOffsetInputStream bIn = new
BodyOffsetInputStream(tmpMsgIn)) {
// Disable line length... This should be handled by the smtp
server
// component and not the parser itself
@@ -433,6 +435,7 @@ public class StoreMessageManager implements MessageManager {
// the file now
// via the TeeInputStream
}
+bufferedOut.close();
int bodyStartOctet = (int) bIn.getBodyStartOffset();
if (bodyStartOctet == -1) {
bodyStartOctet = 0;
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 11/15: JAMES-2817 WaitDelayGenerator jitter could remove time from next iteration
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit d3c5a61f0591a9538be670188c59894c8f5862c8
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 12:31:45 2019 +0100
JAMES-2817 WaitDelayGenerator jitter could remove time from next iteration
---
.../mailbox/events/ErrorHandlingContract.java | 4 ++--
.../james/mailbox/events/WaitDelayGenerator.java | 24 +-
.../mailbox/events/WaitDelayGeneratorTest.java | 8
3 files changed, 20 insertions(+), 16 deletions(-)
diff --git
a/mailbox/api/src/test/java/org/apache/james/mailbox/events/ErrorHandlingContract.java
b/mailbox/api/src/test/java/org/apache/james/mailbox/events/ErrorHandlingContract.java
index c6f4494..d0b3923 100644
---
a/mailbox/api/src/test/java/org/apache/james/mailbox/events/ErrorHandlingContract.java
+++
b/mailbox/api/src/test/java/org/apache/james/mailbox/events/ErrorHandlingContract.java
@@ -167,8 +167,8 @@ interface ErrorHandlingContract extends EventBusContract {
softly.assertThat(timeElapsed).hasSize(4);
long minFirstDelayAfter = 100; // first backOff
-long minSecondDelayAfter = 100; // 200 * jitter factor (200 * 0.5)
-long minThirdDelayAfter = 200; // 400 * jitter factor (400 * 0.5)
+long minSecondDelayAfter = 50; // 50 * jitter factor (50 * 0.5)
+long minThirdDelayAfter = 100; // 100 * jitter factor (100 * 0.5)
softly.assertThat(timeElapsed.get(1))
.isAfterOrEqualTo(timeElapsed.get(0).plusMillis(minFirstDelayAfter));
diff --git
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/WaitDelayGenerator.java
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/WaitDelayGenerator.java
index ff0378b..33a3586 100644
---
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/WaitDelayGenerator.java
+++
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/WaitDelayGenerator.java
@@ -24,7 +24,7 @@ import java.time.Duration;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-
+import com.google.common.primitives.Ints;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
@@ -34,12 +34,14 @@ class WaitDelayGenerator {
return new WaitDelayGenerator(retryBackoff);
}
-private static int randomBetween(int lowest, int highest) {
-Preconditions.checkArgument(lowest <= highest, "lowest always has to
be less than or equals highest");
-if (lowest == highest) {
-return lowest;
+private static Duration randomBetween(Duration base, Duration jitter) {
+Preconditions.checkArgument(!jitter.isNegative(), "jitter value should
always be positive");
+if (jitter.isZero()) {
+return base;
}
-return SECURE_RANDOM.nextInt(highest - lowest) + lowest;
+long maxJitterAsMillis = jitter.toMillis();
+long jitterAsMillis =
SECURE_RANDOM.nextInt(Ints.checkedCast(maxJitterAsMillis * 2)) / 2;
+return base.plusMillis(jitterAsMillis);
}
private static final SecureRandom SECURE_RANDOM = new SecureRandom();
@@ -65,11 +67,13 @@ class WaitDelayGenerator {
if (!shouldDelay(retryCount)) {
return Duration.ZERO;
}
-int exponentialFactor = Double.valueOf(Math.pow(2, retryCount -
1)).intValue();
-int minDelay = exponentialFactor * (int)
retryBackoff.getFirstBackoff().toMillis();
-int maxDelay = Double.valueOf(minDelay + minDelay *
retryBackoff.getJitterFactor()).intValue();
+long exponentialFactor = Double.valueOf(Math.pow(2, retryCount -
1)).longValue();
+Duration minDelay =
retryBackoff.getFirstBackoff().multipliedBy(exponentialFactor);
+Duration jitterDelay = retryBackoff.getFirstBackoff()
+.multipliedBy(Double.valueOf(retryBackoff.getJitterFactor() *
100).intValue())
+.dividedBy(100);
-return Duration.ofMillis(randomBetween(minDelay, maxDelay));
+return randomBetween(minDelay, jitterDelay);
}
private boolean shouldDelay(int retryCount) {
diff --git
a/mailbox/event/event-rabbitmq/src/test/java/org/apache/james/mailbox/events/WaitDelayGeneratorTest.java
b/mailbox/event/event-rabbitmq/src/test/java/org/apache/james/mailbox/events/WaitDelayGeneratorTest.java
index 9cb2d62..e7f1fb6 100644
---
a/mailbox/event/event-rabbitmq/src/test/java/org/apache/james/mailbox/events/WaitDelayGeneratorTest.java
+++
b/mailbox/event/event-rabbitmq/src/test/java/org/apache/james/mailbox/events/WaitDelayGeneratorTest.java
@@ -46,13 +46,13 @@ class WaitDelayGeneratorTest {
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(generator.generateDelay(1)
[james-project] 05/15: JAMES-2816 Avoid copying the whole message when we only need to copy metadata
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 6c20dff5a87926ba0657fec8a185c925ce000c26
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 15:15:34 2019 +0100
JAMES-2816 Avoid copying the whole message when we only need to copy
metadata
---
mailbox/memory/src/test/resources/test.eml | 0
.../mailbox/spamassassin/SpamAssassinListenerTest.java | 4 ++--
.../apache/james/mailbox/store/StoreMessageIdManager.java | 2 +-
.../org/apache/james/mailbox/store/StoreMessageManager.java | 13 ++---
.../org/apache/james/mailbox/store/event/EventFactory.java | 4
.../james/mailbox/store/MessageIdManagerTestSystem.java | 2 +-
6 files changed, 6 insertions(+), 19 deletions(-)
diff --git a/mailbox/memory/src/test/resources/test.eml
b/mailbox/memory/src/test/resources/test.eml
new file mode 100644
index 000..e69de29
diff --git
a/mailbox/plugin/spamassassin/src/test/java/org/apache/james/mailbox/spamassassin/SpamAssassinListenerTest.java
b/mailbox/plugin/spamassassin/src/test/java/org/apache/james/mailbox/spamassassin/SpamAssassinListenerTest.java
index 36922be..72d8b39 100644
---
a/mailbox/plugin/spamassassin/src/test/java/org/apache/james/mailbox/spamassassin/SpamAssassinListenerTest.java
+++
b/mailbox/plugin/spamassassin/src/test/java/org/apache/james/mailbox/spamassassin/SpamAssassinListenerTest.java
@@ -241,7 +241,7 @@ public class SpamAssassinListenerTest {
.randomEventId()
.mailboxSession(MAILBOX_SESSION)
.mailbox(inbox)
-.addMessage(message)
+.addMetaData(message.metaData())
.build();
listener.event(addedEvent);
@@ -257,7 +257,7 @@ public class SpamAssassinListenerTest {
.randomEventId()
.mailboxSession(MAILBOX_SESSION)
.mailbox(mailbox1)
-.addMessage(message)
+.addMetaData(message.metaData())
.build();
listener.event(addedEvent);
diff --git
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
index 1b28585..20a5b0f 100644
---
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
+++
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
@@ -395,7 +395,7 @@ public class StoreMessageIdManager implements
MessageIdManager {
.randomEventId()
.mailboxSession(mailboxSession)
.mailbox(mailboxMapper.findMailboxById(mailboxId))
-.addMessage(copy)
+.addMetaData(copy.metaData())
.build(),
new MailboxIdRegistrationKey(mailboxId))
.block();
diff --git
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
index 5d63d3b..8c32c55 100644
---
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
+++
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
@@ -451,12 +451,12 @@ public class StoreMessageManager implements
MessageManager {
MessageMetaData data = appendMessageToStore(message,
attachments, mailboxSession);
Mailbox mailbox = getMailboxEntity();
-MailboxMessage copy = copyMessage(message);
+
eventBus.dispatch(EventFactory.added()
.randomEventId()
.mailboxSession(mailboxSession)
.mailbox(mailbox)
-.addMessage(copy)
+.addMetaData(message.metaData())
.build(),
new
MailboxIdRegistrationKey(mailbox.getMailboxId()))
.block();
@@ -500,15 +500,6 @@ public class StoreMessageManager implements MessageManager
{
return new SimpleMailboxMessage(messageIdFactory.generate(),
internalDate, size, bodyStartOctet, content, flags, propertyBuilder,
getMailboxEntity().getMailboxId(), attachments);
}
-private MailboxMessage copyMessage(MailboxMessage message) throws
MailboxException {
-return SimpleMailboxMessage
-.from(message)
-.mailboxId(message.getMailboxId())
-.uid(message.getUid())
-.modseq(message.getModSeq())
-.build();
-}
-
@Override
public boolean isWriteable(MailboxSession session) throws MailboxException
{
return storeRightManager.isReadWrite(session, mailbox,
getSharedPermanentFlags(session));
diff --git
a/mailbox/st
[james-project] 04/15: fixup! Avoid blob copy when saving byte arrays
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 088123b1cc97da4b48bc277a04f27604971615be
Author: Gautier DI FOLCO
AuthorDate: Tue Jul 2 11:33:56 2019 +0200
fixup! Avoid blob copy when saving byte arrays
---
.../org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
index c38ea12..c80f23e 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
@@ -134,9 +134,8 @@ public class ObjectStorageBlobsDAO implements BlobStore {
.then(Mono.fromCallable(() ->
blobIdFactory.from(hashingInputStream.hash().toString(;
}
-private Mono save(BucketName bucketName, Blob blob) {
-String containerName = this.containerName.value();
-return Mono.fromCallable(() -> blobStore.putBlob(containerName, blob));
+private Mono save(BucketName bucketName, Blob blob) {
+return Mono.fromRunnable(() -> putBlobFunction.putBlob(blob));
}
@Override
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 12/15: JAMES-2817 GroupRegistration deliver doesn't need to use subscribeWith
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 2338e6461779ec114a9eb13f1da887a2f6030a64
Author: Matthieu Baechler
AuthorDate: Fri Feb 15 14:26:33 2019 +0100
JAMES-2817 GroupRegistration deliver doesn't need to use subscribeWith
---
.../main/java/org/apache/james/mailbox/events/GroupRegistration.java | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
index c704ad3..8d477e7 100644
---
a/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
+++
b/mailbox/event/event-rabbitmq/src/main/java/org/apache/james/mailbox/events/GroupRegistration.java
@@ -40,7 +40,6 @@ import com.rabbitmq.client.Connection;
import reactor.core.Disposable;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
-import reactor.core.publisher.MonoProcessor;
import reactor.core.scheduler.Schedulers;
import reactor.rabbitmq.AcknowledgableDelivery;
import reactor.rabbitmq.BindingSpecification;
@@ -143,9 +142,7 @@ class GroupRegistration implements Registration {
.publishOn(Schedulers.elastic())
.flatMap(any -> Mono.fromRunnable(Throwing.runnable(() ->
runListener(event
.onErrorResume(throwable -> retryHandler.handleRetry(event,
currentRetryCount, throwable))
-.then(Mono.fromRunnable(acknowledgableDelivery::ack))
-.subscribeWith(MonoProcessor.create())
-.then();
+.then(Mono.fromRunnable(acknowledgableDelivery::ack));
}
Mono reDeliver(Event event) {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 13/15: Merge remote-tracking branch 'btellier/improve-blob-store-efficiency-v4'
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 859161869055c9ec79d9ab851ff6a9362885da63
Merge: 78813a8 088123b
Author: Antoine DUPRAT
AuthorDate: Thu Jul 4 11:28:31 2019 +0200
Merge remote-tracking branch 'btellier/improve-blob-store-efficiency-v4'
.../main/java/org/apache/james/blob/api/Store.java | 24 ++
.../james/blob/objectstorage/AESPayloadCodec.java | 5 +
.../blob/objectstorage/DefaultPayloadCodec.java| 9
.../blob/objectstorage/ObjectStorageBlobsDAO.java | 24 --
.../james/blob/objectstorage/PayloadCodec.java | 2 ++
.../apache/james/blob/mail/MimeMessageStore.java | 6 +++---
6 files changed, 57 insertions(+), 13 deletions(-)
diff --cc
server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
index 48840f8,c80f23e..aa209b5
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
@@@ -146,14 -158,8 +159,13 @@@ public class ObjectStorageBlobsDAO impl
"Failed to readBytes blob " + blobId.asString(),
cause);
}
-
}
+@Override
+public Mono deleteBucket(BucketName bucketName) {
+throw new NotImplementedException("not implemented");
+}
+
public void deleteContainer() {
blobStore.deleteContainer(containerName.value());
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 07/15: JAMES-2816 Fix Intelij warnings in SimpleMailboxMessage
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 398594db2e6869c37d8d65357ff82b1f3c3ab60b
Author: Benoit Tellier
AuthorDate: Mon Feb 18 10:12:45 2019 +0700
JAMES-2816 Fix Intelij warnings in SimpleMailboxMessage
---
.../mailbox/store/mail/model/impl/SimpleMailboxMessage.java| 10 +++---
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
index 479053d..34b3b5c 100644
---
a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
+++
b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
@@ -138,13 +138,9 @@ public class SimpleMailboxMessage extends
DelegatingMailboxMessage {
SimpleMailboxMessage simpleMailboxMessage = new
SimpleMailboxMessage(messageId, internalDate, size,
bodyStartOctet, content, flags, propertyBuilder, mailboxId,
attachments.build());
-if (uid.isPresent()) {
-simpleMailboxMessage.setUid(uid.get());
-}
+uid.ifPresent(simpleMailboxMessage::setUid);
+modseq.ifPresent(simpleMailboxMessage::setModSeq);
-if (modseq.isPresent()) {
-simpleMailboxMessage.setModSeq(modseq.get());
-}
return simpleMailboxMessage;
}
}
@@ -218,7 +214,7 @@ public class SimpleMailboxMessage extends
DelegatingMailboxMessage {
PropertyBuilder propertyBuilder, MailboxId
mailboxId) {
this(messageId, internalDate, size, bodyStartOctet,
content, flags,
-propertyBuilder, mailboxId,
ImmutableList.of());
+propertyBuilder, mailboxId, ImmutableList.of());
}
@Override
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] branch master updated (78813a8 -> 2548451)
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git. from 78813a8 JAMES-2771 Fix dirty provisionning for CI benchmarks new 862dd15 Avoid blob copy when saving byte arrays new ed91ea3 Position data length when possible new 30dc0b9 Allow Store.Impl to cary along a byte[] representation new 088123b fixup! Avoid blob copy when saving byte arrays new 8591618 Merge remote-tracking branch 'btellier/improve-blob-store-efficiency-v4' new 6c20dff JAMES-2816 Avoid copying the whole message when we only need to copy metadata new 55257ba JAMES-2816 Use buffered streams to handle mails new 398594d JAMES-2816 Fix Intelij warnings in SimpleMailboxMessage new 02fb91a JAMES-2816 Refactor appendMessage method to make it more readable new 031775a Merge remote-tracking branch 'mbaechler/optimize-append-message' new 87136c3 JAMES-2817 Rename subscribeWorkQueue to consumeWorkQueue and subscribe in start() new 37b0b39 JAMES-2817 Avoid C-like cast syntax in GroupRegistration new d3c5a61 JAMES-2817 WaitDelayGenerator jitter could remove time from next iteration new 2338e64 JAMES-2817 GroupRegistration deliver doesn't need to use subscribeWith new 2548451 Merge remote-tracking branch 'mbaechler/some-small-eventBus-fix' The 15 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../mailbox/events/ErrorHandlingContract.java | 4 +- .../james/mailbox/events/GroupRegistration.java| 25 +- .../james/mailbox/events/WaitDelayGenerator.java | 24 +- .../mailbox/events/WaitDelayGeneratorTest.java | 8 +- .../memory/src/test/resources/test.eml | 0 .../spamassassin/SpamAssassinListenerTest.java | 4 +- .../james/mailbox/store/StoreMessageIdManager.java | 2 +- .../james/mailbox/store/StoreMessageManager.java | 290 - .../james/mailbox/store/event/EventFactory.java| 4 - .../mail/model/impl/SimpleMailboxMessage.java | 10 +- .../mailbox/store/MessageIdManagerTestSystem.java | 2 +- .../main/java/org/apache/james/blob/api/Store.java | 24 +- .../james/blob/objectstorage/AESPayloadCodec.java | 5 + .../blob/objectstorage/DefaultPayloadCodec.java| 9 + .../blob/objectstorage/ObjectStorageBlobsDAO.java | 24 +- .../james/blob/objectstorage/PayloadCodec.java | 2 + .../apache/james/blob/mail/MimeMessageStore.java | 6 +- 17 files changed, 259 insertions(+), 184 deletions(-) copy server/protocols/webadmin/webadmin-cassandra/src/main/java/org/apache/james/webadmin/service/CassandraMigrationService.java => mailbox/memory/src/test/resources/test.eml (100%) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 14/15: Merge remote-tracking branch 'mbaechler/optimize-append-message'
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit 031775a2051f4b9f92dbb6cd2cc7377c8f049ab9 Merge: 8591618 02fb91a Author: Antoine DUPRAT AuthorDate: Thu Jul 4 11:29:09 2019 +0200 Merge remote-tracking branch 'mbaechler/optimize-append-message' mailbox/memory/src/test/resources/test.eml | 0 .../spamassassin/SpamAssassinListenerTest.java | 4 +- .../james/mailbox/store/StoreMessageIdManager.java | 2 +- .../james/mailbox/store/StoreMessageManager.java | 290 - .../james/mailbox/store/event/EventFactory.java| 4 - .../mail/model/impl/SimpleMailboxMessage.java | 10 +- .../mailbox/store/MessageIdManagerTestSystem.java | 2 +- 7 files changed, 171 insertions(+), 141 deletions(-) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 02/15: Position data length when possible
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit ed91ea3e94f57cf14719a199dcd3f15f7ad38f01
Author: Benoit Tellier
AuthorDate: Fri Dec 7 14:47:30 2018 +0700
Position data length when possible
---
.../james/blob/objectstorage/AESPayloadCodec.java | 5 +
.../james/blob/objectstorage/DefaultPayloadCodec.java | 9 +
.../blob/objectstorage/ObjectStorageBlobsDAO.java | 18 ++
.../apache/james/blob/objectstorage/PayloadCodec.java | 2 ++
4 files changed, 26 insertions(+), 8 deletions(-)
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/AESPayloadCodec.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/AESPayloadCodec.java
index 2b0eaee..e19e51d 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/AESPayloadCodec.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/AESPayloadCodec.java
@@ -72,6 +72,11 @@ public class AESPayloadCodec implements PayloadCodec {
}
@Override
+public Payload write(byte[] bytes) {
+return write(new ByteArrayInputStream(bytes));
+}
+
+@Override
public Payload write(InputStream inputStream) {
try (FileBackedOutputStream outputStream = new
FileBackedOutputStream(MAX_BYTES.intValue())) {
outputStream.write(aead.encrypt(IOUtils.toByteArray(inputStream),
EMPTY_ASSOCIATED_DATA));
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/DefaultPayloadCodec.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/DefaultPayloadCodec.java
index 6f56b26..9a6fcc3 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/DefaultPayloadCodec.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/DefaultPayloadCodec.java
@@ -19,6 +19,7 @@
package org.apache.james.blob.objectstorage;
+import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Optional;
@@ -32,6 +33,14 @@ public class DefaultPayloadCodec implements PayloadCodec {
}
@Override
+public Payload write(byte[] bytes) {
+if (bytes.length == 0) {
+return write(new ByteArrayInputStream(bytes));
+}
+return new Payload(Payloads.newByteArrayPayload(bytes),
Optional.of(new Long(bytes.length)));
+}
+
+@Override
public InputStream read(Payload payload) throws IOException {
return payload.getPayload().openStream();
}
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
index fb22164..c38ea12 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
@@ -19,7 +19,6 @@
package org.apache.james.blob.objectstorage;
-import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Optional;
@@ -93,15 +92,18 @@ public class ObjectStorageBlobsDAO implements BlobStore {
}
@Override
-public Mono save(byte[] data) {
+public Mono save(BucketName bucketName, byte[] data) {
+Preconditions.checkNotNull(data);
BlobId blobId = blobIdFactory.forPayload(data);
+Payload payload = payloadCodec.write(data);
Blob blob = blobStore.blobBuilder(blobId.asString())
-.payload(payloadCodec.write(new ByteArrayInputStream(data)))
+.payload(payload.getPayload())
+.contentLength(payload.getLength().orElse(new Long(data.length)))
.build();
return save(bucketName, blob)
-.thenApply(any -> blobId);
+.thenReturn(blobId);
}
@Override
@@ -109,7 +111,7 @@ public class ObjectStorageBlobsDAO implements BlobStore {
Preconditions.checkNotNull(data);
BlobId tmpId = blobIdFactory.randomId();
-return save(data, tmpId)
+return save(bucketName, data, tmpId)
.flatMap(id -> updateBlobId(tmpId, id));
}
@@ -121,18 +123,18 @@ public class ObjectStorageBlobsDAO implements BlobStore {
.thenReturn(to);
}
-private Mono save(InputStream data, BlobId id) {
+private Mono save(BucketName bucketName, InputStream data, BlobId
id) {
HashingInputStream hashingInputStream = new
HashingInputStream(Hashing.sha256(), data);
Payload payload = payloadCodec.write(hashingInputStream);
[james-project] 01/15: Avoid blob copy when saving byte arrays
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 862dd154133317eacffc8e5434083ac0d0452b1d
Author: Benoit Tellier
AuthorDate: Fri Dec 7 14:14:26 2018 +0700
Avoid blob copy when saving byte arrays
We can iterate the in memory byte array first to compute the blobId to use,
saving a blob copy.
---
.../blob/objectstorage/ObjectStorageBlobsDAO.java | 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
index 7bbd2da..fb22164 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
@@ -93,8 +93,15 @@ public class ObjectStorageBlobsDAO implements BlobStore {
}
@Override
-public Mono save(BucketName bucketName, byte[] data) {
-return save(bucketName, new ByteArrayInputStream(data));
+public Mono save(byte[] data) {
+BlobId blobId = blobIdFactory.forPayload(data);
+
+Blob blob = blobStore.blobBuilder(blobId.asString())
+.payload(payloadCodec.write(new ByteArrayInputStream(data)))
+.build();
+
+return save(bucketName, blob)
+.thenApply(any -> blobId);
}
@Override
@@ -121,10 +128,15 @@ public class ObjectStorageBlobsDAO implements BlobStore {
.payload(payload.getPayload())
.build();
-return Mono.fromRunnable(() -> putBlobFunction.putBlob(blob))
+return save(blob)
.then(Mono.fromCallable(() ->
blobIdFactory.from(hashingInputStream.hash().toString(;
}
+private Mono save(Blob blob) {
+String containerName = this.containerName.value();
+return Mono.fromCallable(() -> blobStore.putBlob(containerName, blob));
+}
+
@Override
public Mono readBytes(BucketName bucketName, BlobId blobId) {
return Mono.fromCallable(() -> IOUtils.toByteArray(read(bucketName,
blobId)));
@@ -145,7 +157,6 @@ public class ObjectStorageBlobsDAO implements BlobStore {
"Failed to readBytes blob " + blobId.asString(),
cause);
}
-
}
public void deleteContainer() {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2773) Add glowroot in the docker images
[ https://issues.apache.org/jira/browse/JAMES-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antoine Duprat resolved JAMES-2773. --- Resolution: Fixed Fix Version/s: 3.4.0 merged > Add glowroot in the docker images > - > > Key: JAMES-2773 > URL: https://issues.apache.org/jira/browse/JAMES-2773 > Project: James Server > Issue Type: Bug >Reporter: Rémi Kowalski >Priority: Major > Fix For: 3.4.0 > > > In order to have instrumentation on the JMAP and IMAP transactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 02/03: JAMES-2771 Fix concurrency issues on CI benchmark
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit bc257bc82ab29aad7a1e7e4b2f5f096261a07495
Author: Gautier DI FOLCO
AuthorDate: Wed Jul 3 13:42:18 2019 +0200
JAMES-2771 Fix concurrency issues on CI benchmark
---
JenkinsfileStressTests.groovy | 4
1 file changed, 4 insertions(+)
diff --git a/JenkinsfileStressTests.groovy b/JenkinsfileStressTests.groovy
index 076cbdb..737f690 100644
--- a/JenkinsfileStressTests.groovy
+++ b/JenkinsfileStressTests.groovy
@@ -10,6 +10,10 @@
// It may be used to wait for a service to be available.
pipeline {
+options {
+disableConcurrentBuilds()
+}
+
agent none
stages {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] branch master updated (aec776e -> 78813a8)
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git. from aec776e Merge remote-tracking branch 'btellier/JAMES-2807' new f7cf967 JAMES-2773 Fix CI benchmark glowroot creation new bc257bc JAMES-2771 Fix concurrency issues on CI benchmark new 78813a8 JAMES-2771 Fix dirty provisionning for CI benchmarks The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: JenkinsfileStressTests| 88 --- JenkinsfileStressTests.groovy | 9 + 2 files changed, 9 insertions(+), 88 deletions(-) delete mode 100644 JenkinsfileStressTests - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 03/03: JAMES-2771 Fix dirty provisionning for CI benchmarks
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 78813a88b126936a38180547a089aa5fbc47f113
Author: Gautier DI FOLCO
AuthorDate: Wed Jul 3 14:13:25 2019 +0200
JAMES-2771 Fix dirty provisionning for CI benchmarks
---
JenkinsfileStressTests.groovy | 4
1 file changed, 4 insertions(+)
diff --git a/JenkinsfileStressTests.groovy b/JenkinsfileStressTests.groovy
index 737f690..b621958 100644
--- a/JenkinsfileStressTests.groovy
+++ b/JenkinsfileStressTests.groovy
@@ -50,6 +50,10 @@ pipeline {
stage('Start James') {
steps {
script {
+if (fileExists('/srv/bench-running-docker')) {
+echo 'Last build failed, cleaning
provisionning'
+sh 'sudo btrfs subvolume delete
/srv/bench-running-docker'
+}
sh "cd /srv && sudo btrfs subvolume snapshot
bench-snapshot bench-running-docker"
sh 'docker run -d --name=cassandra -p 9042:9042 -v
/srv/bench-running-docker/cassandra:/var/lib/cassandra cassandra:3.11.3'
sh 'docker run -d --name=elasticsearch -p
9200:9200 -v
/srv/bench-running-docker/elasticsearch:/usr/share/elasticsearch/data/elasticsearch
--env "discovery.type=single-node"
docker.elastic.co/elasticsearch/elasticsearch:6.3.2'
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 01/03: JAMES-2773 Fix CI benchmark glowroot creation
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f7cf9679f0d0437fa5745619251956d5f7e84ad0
Author: Gautier DI FOLCO
AuthorDate: Tue Jul 2 15:00:08 2019 +0200
JAMES-2773 Fix CI benchmark glowroot creation
---
JenkinsfileStressTests| 88 ---
JenkinsfileStressTests.groovy | 1 +
2 files changed, 1 insertion(+), 88 deletions(-)
diff --git a/JenkinsfileStressTests b/JenkinsfileStressTests
deleted file mode 100644
index 9bf9425..000
--- a/JenkinsfileStressTests
+++ /dev/null
@@ -1,88 +0,0 @@
-// This file should respect the Jenkinsfile format describe here:
-// https://jenkins.io/doc/book/pipeline/jenkinsfile/
-//
-// It may be used by any Jenkins instance you own.
-//
-// In order to work properly, it requires the following parameters:
-// - SENARIO: the Gatling scenrio you want to play (ex.
com.linagora.gatling.imap.scenario.ImapSimpleScenario)
-
-// Method in order to retry a command.
-// It may be used to wait for a service to be available.
-def maxRetries = 120
-def waitForCommandSuccess(command, maxRetries) {
-def tries = 0;
-while (tries++ < maxRetries) {
-try {
-sh command
-return true
-} catch (Exception e) {
-if (tries >= maxRetries) {
-throw new Exception("${maxRetries} tries exceeded. ${command}
failed.", e)
-}
-sleep 1 SECONDS
-}
-}
-}
-
-pipeline {
-agent { label 'my-slave' }
-
-tools {
-maven 'maven'
-}
-
-stages {
-stage('Compile') {
-steps {
-sh "mvn clean install -T1C -DskipTests"
-}
-}
-stage('Build image') {
-steps {
-script {
-docker.withTool('docker') {
-sh "cp
server/container/guice/cassandra-rabbitmq-guice/target/james-server-cassandra-rabbitmq-guice.jar
dockerfiles/run/guice/cassandra-rabbitmq/destination"
-sh "cp -r
server/container/guice/cassandra-rabbitmq-guice/target/james-server-cassandra-rabbitmq-guice.lib
dockerfiles/run/guice/cassandra-rabbitmq/destination"
-sh "cp
server/container/cli/target/james-server-cli.jar
dockerfiles/run/guice/cassandra-rabbitmq/destination"
-sh "cp -r
server/container/cli/target/james-server-cli.lib
dockerfiles/run/guice/cassandra-rabbitmq/destination"
-sh "docker build -t james_run
dockerfiles/run/guice/cassandra-rabbitmq"
-}
-}
-}
-}
-stage('Start James') {
-steps {
-script {
-docker.withTool('docker') {
-sh "docker run -d --name=cassandra cassandra:3.11.3"
-sh "docker run -d --name=rabbitmq
rabbitmq:3.7.7-management"
-sh "docker run -d --name=elasticsearch --env
'discovery.type=single-node'
docker.elastic.co/elasticsearch/elasticsearch:6.3.2"
-
-sh "docker run -d --hostname HOSTNAME -p 25:25 -p
8081:80 -p 110:110 -p 143:143 -p 465:465 -p 587:587 -p 993:993 --link
cassandra:cassandra --link rabbitmq:rabbitmq --link elasticsearch:elasticsearch
--name james_run -t james_run"
-try {
-def jamesCliWithOptions = 'java -jar
/root/james-cli.jar -h 127.0.0.1'
-waitForCommandSuccess("docker exec james_run
${jamesCliWithOptions} listusers", maxRetries)
-} catch (Exception e) {
-sh "docker logs james_run"
-}
-}
-}
-}
-}
-stage('Run Gatling test') {
-steps {
-build job: 'Gatling job', parameters: [[$class:
'StringParameterValue', name: 'SBT_ACTION', value: "gatling:testOnly
${params.SCENARIO}"]]
-}
-}
-}
-
-post {
-always {
-script {
-docker.withTool('docker') {
-sh "docker rm -f cassandra rabbitmq elasticsearch
james_run"
-}
-}
-}
-}
-}
diff --git a/JenkinsfileStressTests.groovy b/JenkinsfileStressTests.groovy
index f9fb851..076cbdb 100644
--- a/JenkinsfileStressTests.groovy
+++ b/JenkinsfileStressTests.groovy
@@ -38,6 +38,7 @@ pipeline {
sh "cp
server/container/cli/target/james-server-cli.jar
dockerfiles/run/guice/cassandra-rabbitmq/destination"
sh "cp -r
server/container/cli/target/james-server-cli.lib
dockerfiles/run/guice/cassandra-rabbitmq/destination"
sh 'cp
server/protocols/jmap-integration-testing/rabbitmq-jmap-integra
[jira] [Created] (JAMES-2817) Various EventBus fixes
Gautier DI FOLCO created JAMES-2817: --- Summary: Various EventBus fixes Key: JAMES-2817 URL: https://issues.apache.org/jira/browse/JAMES-2817 Project: James Server Issue Type: Improvement Reporter: Gautier DI FOLCO -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-2816) Append messages optimization
Gautier DI FOLCO created JAMES-2816: --- Summary: Append messages optimization Key: JAMES-2816 URL: https://issues.apache.org/jira/browse/JAMES-2816 Project: James Server Issue Type: Improvement Reporter: Gautier DI FOLCO We tend to copy the whole message when only meta-data are needed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Duprat closed JAMES-2806.
-
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2807) DeletedMessageMetadataVault API + contract +memory
[
https://issues.apache.org/jira/browse/JAMES-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Duprat resolved JAMES-2807.
---
Resolution: Fixed
Fix Version/s: 3.4.0
merged
> DeletedMessageMetadataVault API + contract +memory
> --
>
> Key: JAMES-2807
> URL: https://issues.apache.org/jira/browse/JAMES-2807
> Project: James Server
> Issue Type: Sub-task
> Components: deletedMessageVault
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> We need a sub API for storing per Container metadata of Deleted Messages.
> Here is the API:
> {code:java}
> class StorageInformation {
> private final BucketName bucketName;
> private final BlobId blobId;
> }
> class DeletedMessageWithStorageInformation {
> private final DeletedMessage deletedmessage;
> private final StorageInformation storageInformation;
> }
> interface DeletedMessageMetadataVault {
> Publisher store(BucketName, DeletedMessageWithStorageInformation);
> Publisher removeBucket(BucketName);
> Publisher remove(BucketName, User, MessageId);
> Publisher retrieveStorageInformation(User, MessageId);
> Publisher listMessages(BucketName,
> User);
> Publisher listBuckets();
> }
> {code}
> You will provide the API + contract in
> `mailbox/plugin/deleted-messages-vault`.
> You will provide a memory implementation of it in
> `mailbox/plugin/deleted-messages-vault-memory`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2807) DeletedMessageMetadataVault API + contract +memory
[
https://issues.apache.org/jira/browse/JAMES-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Duprat closed JAMES-2807.
-
> DeletedMessageMetadataVault API + contract +memory
> --
>
> Key: JAMES-2807
> URL: https://issues.apache.org/jira/browse/JAMES-2807
> Project: James Server
> Issue Type: Sub-task
> Components: deletedMessageVault
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> We need a sub API for storing per Container metadata of Deleted Messages.
> Here is the API:
> {code:java}
> class StorageInformation {
> private final BucketName bucketName;
> private final BlobId blobId;
> }
> class DeletedMessageWithStorageInformation {
> private final DeletedMessage deletedmessage;
> private final StorageInformation storageInformation;
> }
> interface DeletedMessageMetadataVault {
> Publisher store(BucketName, DeletedMessageWithStorageInformation);
> Publisher removeBucket(BucketName);
> Publisher remove(BucketName, User, MessageId);
> Publisher retrieveStorageInformation(User, MessageId);
> Publisher listMessages(BucketName,
> User);
> Publisher listBuckets();
> }
> {code}
> You will provide the API + contract in
> `mailbox/plugin/deleted-messages-vault`.
> You will provide a memory implementation of it in
> `mailbox/plugin/deleted-messages-vault-memory`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2806) Adding bucket capability into BlobStore
[
https://issues.apache.org/jira/browse/JAMES-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Duprat resolved JAMES-2806.
---
Resolution: Fixed
Fix Version/s: 3.4.0
merged
> Adding bucket capability into BlobStore
> ---
>
> Key: JAMES-2806
> URL: https://issues.apache.org/jira/browse/JAMES-2806
> Project: James Server
> Issue Type: Sub-task
> Components: Blob
>Reporter: Tellier Benoit
>Priority: Major
> Fix For: 3.4.0
>
>
> You should update current BlobStore API to be able to customize the way we
> storeBlob:
> - We want to store a blob in a specified bucket
> {code:java}
> public interface BlobStore {
> Publisher save(BucketName, byte[] data);
> Publisher save(BucketName, InputStream data);
> Publisher readBytes(BucketName, BlobId blobId);
> InputStream read(BucketName, BlobId blobId);
> }
> {code}
> And to implement the feature of deleting buckets exceed retention time, we
> need to have another additional method of deleting bucket:
> {code:java}
> public interface BlobStore {
> Publisher delete(BucketName bucket);
> }
> {code}
> Scope: API change, contract test, memory + object storage implementation -
> cassandra will be done in a later task
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 07/13: JAMES-2807 Create POJOs for DeletedMessageMetadataVault API
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 9ebaa14a0ace03e0b2b19b7cee18b15e95376183
Author: Benoit Tellier
AuthorDate: Tue Jun 25 11:21:36 2019 +0700
JAMES-2807 Create POJOs for DeletedMessageMetadataVault API
---
.../deleted-messages-vault-blobstore/pom.xml | 86 ++
.../DeletedMessageWithStorageInformation.java | 61 +++
.../org/apache/james/vault/StorageInformation.java | 64
.../vault/DeletedMessageVaultMetadataFixture.java | 30
.../DeletedMessageWithStorageInformationTest.java | 47
.../apache/james/vault/StorageInformationTest.java | 47
mailbox/pom.xml| 4 +
7 files changed, 339 insertions(+)
diff --git a/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
b/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
new file mode 100644
index 000..5575f57
--- /dev/null
+++ b/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
@@ -0,0 +1,86 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+apache-james-mailbox
+org.apache.james
+3.4.0-SNAPSHOT
+../../pom.xml
+
+
+
apache-james-mailbox-deleted-messages-vault-blobstore
+Apache James :: Mailbox :: Plugin :: Deleted Messages Vault ::
BlobStore
+Apache James Mailbox Deleted Messages Vault implementation on
top of BlobStore
+
+
+
+${james.groupId}
+apache-james-mailbox-api
+test-jar
+test
+
+
+${james.groupId}
+
apache-james-mailbox-deleted-messages-vault
+
+
+${james.groupId}
+
apache-james-mailbox-deleted-messages-vault
+test-jar
+test
+
+
+${james.groupId}
+apache-james-mailbox-memory
+test
+
+
+${james.groupId}
+apache-james-mailbox-memory
+test
+test-jar
+
+
+${james.groupId}
+blob-api
+
+
+nl.jqno.equalsverifier
+equalsverifier
+test
+
+
+org.assertj
+assertj-core
+test
+
+
+org.junit.jupiter
+junit-jupiter-engine
+test
+
+
+org.junit.platform
+junit-platform-launcher
+test
+
+
+
+
\ No newline at end of file
diff --git
a/mailbox/plugin/deleted-messages-vault-blobstore/src/main/java/org/apache/james/vault/DeletedMessageWithStorageInformation.java
b/mailbox/plugin/deleted-messages-vault-blobstore/src/main/java/org/apache/james/vault/DeletedMessageWithStorageInformation.java
new file mode 100644
index 000..669b3a6
--- /dev/null
+++
b/mailbox/plugin/deleted-messages-vault-blobstore/src/main/java/org/apache/james/vault/DeletedMessageWithStorageInformation.java
@@ -0,0 +1,61 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.vault;
+
+import java.util.Objects;
+
+import com.google.common.base.Preconditions;
+
+public class DeletedMessageWithStorageInformation {
+private final DeletedMessage deletedmessage;
+private final StorageInformation storageInformation;
+
+public DeletedMessageWithStorageInformation(DeletedMessage deletedmessage,
StorageInformation storageInformation) {
+Preconditions.checkNotNull(deletedmessage);
+
[james-project] 05/13: JAMES-2806 add a metric for deleteBucket method and test in its contract
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 6ec43fb87bd37317da03574c3a5939aa50263563
Author: Rene Cordier
AuthorDate: Tue Jul 2 10:35:48 2019 +0700
JAMES-2806 add a metric for deleteBucket method and test in its contract
---
.../org/apache/james/blob/api/MetricableBlobStore.java | 4 +++-
.../apache/james/blob/api/MetricableBlobStoreContract.java | 14 ++
.../apache/james/blob/cassandra/CassandraBlobsDAOTest.java | 7 +++
.../blob/objectstorage/ObjectStorageBlobsDAOTest.java | 7 +++
4 files changed, 31 insertions(+), 1 deletion(-)
diff --git
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
index 89548d8..11fd997 100644
---
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
+++
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
@@ -36,6 +36,7 @@ public class MetricableBlobStore implements BlobStore {
static final String SAVE_INPUT_STREAM_TIMER_NAME =
BLOB_STORE_METRIC_PREFIX + "saveInputStream";
static final String READ_BYTES_TIMER_NAME = BLOB_STORE_METRIC_PREFIX +
"readBytes";
static final String READ_TIMER_NAME = BLOB_STORE_METRIC_PREFIX + "read";
+static final String DELETE_TIMER_NAME = BLOB_STORE_METRIC_PREFIX +
"delete";
private final MetricFactory metricFactory;
private final BlobStore blobStoreImpl;
@@ -73,6 +74,7 @@ public class MetricableBlobStore implements BlobStore {
@Override
public Mono deleteBucket(BucketName bucketName) {
-return blobStoreImpl.deleteBucket(bucketName);
+return metricFactory
+.runPublishingTimerMetric(DELETE_TIMER_NAME, () ->
blobStoreImpl.deleteBucket(bucketName));
}
}
diff --git
a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/MetricableBlobStoreContract.java
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/MetricableBlobStoreContract.java
index 74a30ff..983c516 100644
---
a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/MetricableBlobStoreContract.java
+++
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/MetricableBlobStoreContract.java
@@ -19,6 +19,7 @@
package org.apache.james.blob.api;
+import static org.apache.james.blob.api.MetricableBlobStore.DELETE_TIMER_NAME;
import static
org.apache.james.blob.api.MetricableBlobStore.READ_BYTES_TIMER_NAME;
import static org.apache.james.blob.api.MetricableBlobStore.READ_TIMER_NAME;
import static
org.apache.james.blob.api.MetricableBlobStore.SAVE_BYTES_TIMER_NAME;
@@ -101,4 +102,17 @@ public interface MetricableBlobStoreContract extends
BlobStoreContract {
assertThat(metricsTestExtension.getMetricFactory().executionTimesFor(READ_TIMER_NAME))
.hasSize(2);
}
+
+@Test
+default void deleteBucketShouldPublishDeleteTimerMetrics() {
+BucketName bucketName = BucketName.of("custom");
+testee().save(BucketName.DEFAULT, BYTES_CONTENT).block();
+testee().save(bucketName, BYTES_CONTENT).block();
+
+testee().deleteBucket(BucketName.DEFAULT);
+testee().deleteBucket(bucketName);
+
+
assertThat(metricsTestExtension.getMetricFactory().executionTimesFor(DELETE_TIMER_NAME))
+.hasSize(2);
+}
}
\ No newline at end of file
diff --git
a/server/blob/blob-cassandra/src/test/java/org/apache/james/blob/cassandra/CassandraBlobsDAOTest.java
b/server/blob/blob-cassandra/src/test/java/org/apache/james/blob/cassandra/CassandraBlobsDAOTest.java
index 758ad7e..23e3375 100644
---
a/server/blob/blob-cassandra/src/test/java/org/apache/james/blob/cassandra/CassandraBlobsDAOTest.java
+++
b/server/blob/blob-cassandra/src/test/java/org/apache/james/blob/cassandra/CassandraBlobsDAOTest.java
@@ -35,6 +35,7 @@ import org.apache.james.blob.api.MetricableBlobStore;
import org.apache.james.blob.api.MetricableBlobStoreContract;
import org.apache.james.util.ZeroedInputStream;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -70,6 +71,12 @@ public class CassandraBlobsDAOTest implements
MetricableBlobStoreContract {
return new HashBlobId.Factory();
}
+@Override
+@Disabled("JAMES-2806: delete bucket not implemented yet for Cassandra")
+public void deleteBucketShouldPublishDeleteTimerMetrics() {
+
+}
+
@Test
void readBytesShouldReturnSplitSavedDataByChunk() {
String longString = Strings.repeat("0123456789\n",
MULTIPLE_CHUNK_SIZE);
diff --git
a/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOTest.java
b/server
[james-project] branch master updated (9d244d5 -> aec776e)
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from 9d244d5 JAMES-2813 small fixes on TaskManagerContract
new 7257bc8 JAMES-2806 add deleteBucket method in BlobStore
new 53d3ed7 JAMES-2806 implement buckets with memory BlobStore
new 2672ea9 JAMES-2806 add contract for BlobStore with custom buckets
new 1a7e058 JAMES-2806 Add deleteBucket implementation for memory and its
contract
new 6ec43fb JAMES-2806 add a metric for deleteBucket method and test in
its contract
new 2a2af71 JAMES-2806 add tests for deleteBucket with UnionBlobStore
new 40292be Merge remote-tracking branch
'rcordier/james-2806-bucket-name-memory'
new 9ebaa14 JAMES-2807 Create POJOs for DeletedMessageMetadataVault API
new c7177df JAMES-2807 DeletedMessageMetadataVault API and memory skeleton
new 5063bfc JAMES-2807 Implement Memory DeletedMessageMetadataVault and
related contract
new 092e127 JAMES-2807 Simplify maven structure for the vault
new e7b5683 JAMES-2807 Make it clear that DeletedMessageMetadataVault do
not actually store buckets
new aec776e Merge remote-tracking branch 'btellier/JAMES-2807'
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
mailbox/plugin/deleted-messages-vault/pom.xml | 4 +
.../MemoryDeletedMessageMetadataVault.java | 115
.../DeletedMessageMetadataVault.java} | 21 +--
.../DeletedMessageWithStorageInformation.java} | 51 +++---
.../james/vault/metadata/StorageInformation.java | 44 ++---
.../MemoryDeletedMessageMetadataVaultTest.java | 19 +-
.../DeletedMessageMetadataVaultContract.java | 199 +
.../DeletedMessageVaultMetadataFixture.java| 23 +--
.../DeletedMessageWithStorageInformationTest.java | 24 ++-
.../vault/metadata/StorageInformationTest.java | 30 ++--
mailbox/pom.xml| 3 +
.../java/org/apache/james/blob/api/BlobStore.java | 2 +
.../apache/james/blob/api/MetricableBlobStore.java | 7 +
.../james/blob/api/BucketBlobStoreContract.java| 142 +++
.../blob/api/MetricableBlobStoreContract.java | 14 ++
.../james/blob/cassandra/CassandraBlobsDAO.java| 6 +
.../blob/cassandra/CassandraBlobsDAOTest.java | 7 +
.../apache/james/blob/memory/MemoryBlobStore.java | 44 +++--
.../james/blob/memory/MemoryBlobStoreTest.java | 3 +-
.../blob/objectstorage/ObjectStorageBlobsDAO.java | 6 +
.../objectstorage/ObjectStorageBlobsDAOTest.java | 7 +
.../apache/james/blob/union/UnionBlobStore.java| 7 +
.../james/blob/union/UnionBlobStoreTest.java | 49 +
23 files changed, 692 insertions(+), 135 deletions(-)
create mode 100644
mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVault.java
copy
mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/{DeletedMessageVault.java
=> metadata/DeletedMessageMetadataVault.java} (68%)
copy
mailbox/plugin/{quota-search/src/main/java/org/apache/james/quota/search/Limit.java
=>
deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/DeletedMessageWithStorageInformation.java}
(56%)
copy
server/data/data-library/src/main/java/org/apache/james/dlp/eventsourcing/commands/ClearCommand.java
=>
mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/StorageInformation.java
(62%)
copy
server/mailrepository/mailrepository-memory/src/test/java/org/apache/james/mailrepository/memory/MemoryMailRepositoryTest.java
=>
mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVaultTest.java
(68%)
create mode 100644
mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageMetadataVaultContract.java
copy
server/container/guice/blob-memory-guice/src/main/java/org/apache/james/modules/BlobMemoryModule.java
=>
mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageVaultMetadataFixture.java
(69%)
copy
server/data/data-jmap/src/test/java/org/apache/james/jmap/api/filtering/impl/DefineRulesCommandTest.java
=>
mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageWithStorageInformationTest.java
(67%)
copy mailet/api/src/test/java/org/apache/mailet/AttributeNameTest.java =>
mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/StorageInformationTest.java
(70%)
create mode 100644
server/blob/blob-api/src/test/java/org
[james-project] 12/13: Merge remote-tracking branch 'rcordier/james-2806-bucket-name-memory'
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit 40292bec751517ba5c64a0026dea219a9e939e43 Merge: 9d244d5 2a2af71 Author: Antoine DUPRAT AuthorDate: Thu Jul 4 10:32:39 2019 +0200 Merge remote-tracking branch 'rcordier/james-2806-bucket-name-memory' .../java/org/apache/james/blob/api/BlobStore.java | 2 + .../apache/james/blob/api/MetricableBlobStore.java | 7 + .../james/blob/api/BucketBlobStoreContract.java| 142 + .../blob/api/MetricableBlobStoreContract.java | 14 ++ .../james/blob/cassandra/CassandraBlobsDAO.java| 6 + .../blob/cassandra/CassandraBlobsDAOTest.java | 7 + .../apache/james/blob/memory/MemoryBlobStore.java | 44 +-- .../james/blob/memory/MemoryBlobStoreTest.java | 3 +- .../blob/objectstorage/ObjectStorageBlobsDAO.java | 6 + .../objectstorage/ObjectStorageBlobsDAOTest.java | 7 + .../apache/james/blob/union/UnionBlobStore.java| 7 + .../james/blob/union/UnionBlobStoreTest.java | 49 +++ 12 files changed, 282 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 13/13: Merge remote-tracking branch 'btellier/JAMES-2807'
This is an automated email from the ASF dual-hosted git repository. aduprat pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit aec776ebb4bce831dd4ae08ab38e28b0b9d6c4e7 Merge: 40292be e7b5683 Author: Antoine DUPRAT AuthorDate: Thu Jul 4 10:33:15 2019 +0200 Merge remote-tracking branch 'btellier/JAMES-2807' mailbox/plugin/deleted-messages-vault/pom.xml | 4 + .../MemoryDeletedMessageMetadataVault.java | 115 .../metadata/DeletedMessageMetadataVault.java | 39 .../DeletedMessageWithStorageInformation.java | 63 +++ .../james/vault/metadata/StorageInformation.java | 64 +++ .../MemoryDeletedMessageMetadataVaultTest.java | 38 .../DeletedMessageMetadataVaultContract.java | 199 + .../DeletedMessageVaultMetadataFixture.java| 31 .../DeletedMessageWithStorageInformationTest.java | 48 + .../vault/metadata/StorageInformationTest.java | 47 + mailbox/pom.xml| 3 + 11 files changed, 651 insertions(+) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 06/13: JAMES-2806 add tests for deleteBucket with UnionBlobStore
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 2a2af7101ec83fa49173412499ea569f84200b36
Author: Rene Cordier
AuthorDate: Tue Jul 2 10:39:04 2019 +0700
JAMES-2806 add tests for deleteBucket with UnionBlobStore
---
.../james/blob/union/UnionBlobStoreTest.java | 39 ++
1 file changed, 39 insertions(+)
diff --git
a/server/blob/blob-union/src/test/java/org/apache/james/blob/union/UnionBlobStoreTest.java
b/server/blob/blob-union/src/test/java/org/apache/james/blob/union/UnionBlobStoreTest.java
index a44fd0d..1789129 100644
---
a/server/blob/blob-union/src/test/java/org/apache/james/blob/union/UnionBlobStoreTest.java
+++
b/server/blob/blob-union/src/test/java/org/apache/james/blob/union/UnionBlobStoreTest.java
@@ -489,4 +489,43 @@ class UnionBlobStoreTest implements BlobStoreContract {
assertThat(pushBackIS)
.hasSameContentAs(new ByteArrayInputStream(new byte[0]));
}
+
+@Test
+void deleteBucketShouldDeleteBothCurrentAndLegacyBuckets() {
+BlobId legacyBlobId = legacyBlobStore.save(BucketName.DEFAULT,
BLOB_CONTENT).block();
+BlobId currentBlobId = currentBlobStore.save(BucketName.DEFAULT,
BLOB_CONTENT).block();
+
+unionBlobStore.deleteBucket(BucketName.DEFAULT).block();
+
+assertThatThrownBy(() -> legacyBlobStore.readBytes(BucketName.DEFAULT,
legacyBlobId).block())
+.isInstanceOf(ObjectStoreException.class);
+assertThatThrownBy(() ->
currentBlobStore.readBytes(BucketName.DEFAULT, currentBlobId).block())
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+void deleteBucketShouldDeleteCurrentBucketEvenWhenLegacyDoesNotExist() {
+BlobId currentBlobId = currentBlobStore.save(BucketName.DEFAULT,
BLOB_CONTENT).block();
+
+unionBlobStore.deleteBucket(BucketName.DEFAULT).block();
+
+assertThatThrownBy(() ->
currentBlobStore.readBytes(BucketName.DEFAULT, currentBlobId).block())
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+void deleteBucketShouldDeleteLegacyBucketEvenWhenCurrentDoesNotExist() {
+BlobId legacyBlobId = legacyBlobStore.save(BucketName.DEFAULT,
BLOB_CONTENT).block();
+
+unionBlobStore.deleteBucket(BucketName.DEFAULT).block();
+
+assertThatThrownBy(() -> legacyBlobStore.readBytes(BucketName.DEFAULT,
legacyBlobId).block())
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+void deleteBucketShouldNotThrowWhenCurrentAndLegacyBucketsDoNotExist() {
+assertThatCode(() ->
unionBlobStore.deleteBucket(BucketName.DEFAULT).block())
+.doesNotThrowAnyException();
+}
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 11/13: JAMES-2807 Make it clear that DeletedMessageMetadataVault do not actually store buckets
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e7b56830237716782a7dac3dc2476a44452f379f
Author: Benoit Tellier
AuthorDate: Mon Jul 1 16:00:12 2019 +0700
JAMES-2807 Make it clear that DeletedMessageMetadataVault do not actually
store buckets
It only exposes primitives to take actions on metadata related to a bucket
used for the
DeletedMessageVault
---
.../metadata/MemoryDeletedMessageMetadataVault.java| 6 +++---
.../vault/metadata/DeletedMessageMetadataVault.java| 4 ++--
.../metadata/DeletedMessageMetadataVaultContract.java | 18 +-
3 files changed, 14 insertions(+), 14 deletions(-)
diff --git
a/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVault.java
b/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVault.java
index ccf5fc6..59ede6d 100644
---
a/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVault.java
+++
b/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/memory/metadata/MemoryDeletedMessageMetadataVault.java
@@ -62,7 +62,7 @@ public class MemoryDeletedMessageMetadataVault implements
DeletedMessageMetadata
}
@Override
-public Publisher removeBucket(BucketName bucketName) {
+public Publisher removeMetadataRelatedToBucket(BucketName
bucketName) {
return Mono.fromRunnable(() -> {
synchronized (table) {
table.row(bucketName).clear();
@@ -81,7 +81,7 @@ public class MemoryDeletedMessageMetadataVault implements
DeletedMessageMetadata
@Override
public Publisher retrieveStorageInformation(User user,
MessageId messageId) {
-return Flux.from(listBuckets())
+return Flux.from(listRelatedBuckets())
.concatMap(bucket -> {
synchronized (table) {
return Mono.justOrEmpty(userVault(bucket,
user).get(messageId));
@@ -102,7 +102,7 @@ public class MemoryDeletedMessageMetadataVault implements
DeletedMessageMetadata
}
@Override
-public Publisher listBuckets() {
+public Publisher listRelatedBuckets() {
synchronized (table) {
return Flux.fromIterable(ImmutableSet.copyOf(table.rowKeySet()));
}
diff --git
a/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/DeletedMessageMetadataVault.java
b/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/DeletedMessageMetadataVault.java
index 25b5a4a..3aab7aa 100644
---
a/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/DeletedMessageMetadataVault.java
+++
b/mailbox/plugin/deleted-messages-vault/src/main/java/org/apache/james/vault/metadata/DeletedMessageMetadataVault.java
@@ -27,7 +27,7 @@ import org.reactivestreams.Publisher;
public interface DeletedMessageMetadataVault {
Publisher store(DeletedMessageWithStorageInformation deletedMessage);
-Publisher removeBucket(BucketName bucketName);
+Publisher removeMetadataRelatedToBucket(BucketName bucketName);
Publisher remove(BucketName bucketName, User user, MessageId
messageId);
@@ -35,5 +35,5 @@ public interface DeletedMessageMetadataVault {
Publisher listMessages(BucketName
bucketName, User user);
-Publisher listBuckets();
+Publisher listRelatedBuckets();
}
diff --git
a/mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageMetadataVaultContract.java
b/mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageMetadataVaultContract.java
index 3b85bc5..0052f0e 100644
---
a/mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageMetadataVaultContract.java
+++
b/mailbox/plugin/deleted-messages-vault/src/test/java/org/apache/james/vault/metadata/DeletedMessageMetadataVaultContract.java
@@ -79,7 +79,7 @@ public interface DeletedMessageMetadataVaultContract {
@Test
default void listBucketsShouldBeEmptyWhenNoMessageInserted() {
-Stream messages =
Flux.from(metadataVault().listBuckets()).toStream();
+Stream messages =
Flux.from(metadataVault().listRelatedBuckets()).toStream();
assertThat(messages).isEmpty();
}
@@ -88,7 +88,7 @@ public interface DeletedMessageMetadataVaultContract {
Mono.from(metadataVault().store(DELETED_MESSAGE)).block();
Mono.from(metadataVault().store(DELETED_MESSAGE_2_OTHER_BUCKET)).block();
-Stream messages =
Flux.from(metadataVault().listBuckets()).toStream();
+Stream messages =
Flux.from(metadataVault().listRelatedBuckets()).toStream();
assertT
[james-project] 10/13: JAMES-2807 Simplify maven structure for the vault
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 092e127f8f324b1f51bce87b58882856c517227b
Author: Benoit Tellier
AuthorDate: Mon Jul 1 14:17:53 2019 +0700
JAMES-2807 Simplify maven structure for the vault
Define the metadata storage API within the vault API.
As the memory implementation is shipped as part of the vault API, include
the memory implementation for metadata storage there too.
---
.../pom.xml| 87 --
.../deleted-messages-vault-blobstore/pom.xml | 86 -
mailbox/plugin/deleted-messages-vault/pom.xml | 4 +
.../MemoryDeletedMessageMetadataVault.java | 5 +-
.../metadata}/DeletedMessageMetadataVault.java | 2 +-
.../DeletedMessageWithStorageInformation.java | 4 +-
.../james/vault/metadata}/StorageInformation.java | 2 +-
.../MemoryDeletedMessageMetadataVaultTest.java | 4 +-
.../DeletedMessageMetadataVaultContract.java | 9 ++-
.../DeletedMessageVaultMetadataFixture.java| 2 +-
.../DeletedMessageWithStorageInformationTest.java | 5 +-
.../vault/metadata}/StorageInformationTest.java| 6 +-
mailbox/pom.xml| 2 -
pom.xml| 11 ---
14 files changed, 28 insertions(+), 201 deletions(-)
diff --git a/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
deleted file mode 100644
index faf54ee..000
--- a/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
+++ /dev/null
@@ -1,87 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
-4.0.0
-
-
-apache-james-mailbox
-org.apache.james
-3.4.0-SNAPSHOT
-../../pom.xml
-
-
-
apache-james-mailbox-deleted-messages-vault-blobstore-memory
-Apache James :: Mailbox :: Plugin :: Deleted Messages Vault ::
BlobStore :: Memory
-Memory implementation of the metadata storage API required by
the Deleted Messages Vault implementation on top of BlobStore
-
-
-
-${james.groupId}
-apache-james-mailbox-api
-test-jar
-test
-
-
-${james.groupId}
-
apache-james-mailbox-deleted-messages-vault
-test-jar
-test
-
-
-${james.groupId}
-
apache-james-mailbox-deleted-messages-vault-blobstore
-
-
-${james.groupId}
-
apache-james-mailbox-deleted-messages-vault-blobstore
-test-jar
-test
-
-
-${james.groupId}
-apache-james-mailbox-memory
-test
-
-
-${james.groupId}
-apache-james-mailbox-memory
-test
-test-jar
-
-
-com.google.guava
-guava
-
-
-org.assertj
-assertj-core
-test
-
-
-org.junit.jupiter
-junit-jupiter-engine
-test
-
-
-org.junit.platform
-junit-platform-launcher
-test
-
-
-
-
\ No newline at end of file
diff --git a/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
b/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
deleted file mode 100644
index 5575f57..000
--- a/mailbox/plugin/deleted-messages-vault-blobstore/pom.xml
+++ /dev/null
@@ -1,86 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
-4.0.0
-
-
-apache-james-mailbox
-org.apache.james
-3.4.0-SNAPSHOT
-../../pom.xml
-
-
-
apache-james-mailbox-deleted-messages-vault-blobstore
-Apache James :: Mailbox :: Plugin :: Deleted Messages Vault ::
BlobStore
-Apache James Mailbox Deleted Messages Vault implementation on
top of BlobStore
-
-
-
-${james.groupId}
-apache-james-mailbox-api
-test-jar
-test
-
-
-${james.groupId}
-
apache-james-mailbox-deleted-messages-vault
-
-
-${james.groupId}
-
apache-james-mailbox-deleted-messages-vault
-test-jar
-test
-
-
-${james.groupId}
-apache-james-mailbox-memory
-test
-
-
-${james.groupId}
-apache-james-mai
[james-project] 08/13: JAMES-2807 DeletedMessageMetadataVault API and memory skeleton
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit c7177df2f5be728270516db4580cba6e4acefa13
Author: Benoit Tellier
AuthorDate: Tue Jun 25 11:31:16 2019 +0700
JAMES-2807 DeletedMessageMetadataVault API and memory skeleton
---
.../pom.xml| 40 +++
.../vault/MemoryDeletedMessageMetadataVault.java | 58 ++
.../james/vault/DeletedMessageMetadataVault.java | 39 +++
mailbox/pom.xml| 1 +
pom.xml| 11
5 files changed, 149 insertions(+)
diff --git a/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
new file mode 100644
index 000..d5ae54e
--- /dev/null
+++ b/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
@@ -0,0 +1,40 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+apache-james-mailbox
+org.apache.james
+3.4.0-SNAPSHOT
+../../pom.xml
+
+
+
apache-james-mailbox-deleted-messages-vault-blobstore-memory
+Apache James :: Mailbox :: Plugin :: Deleted Messages Vault ::
BlobStore :: Memory
+Memory implementation of the metadata storage API required by
the Deleted Messages Vault implementation on top of BlobStore
+
+
+
+org.apache.james
+
apache-james-mailbox-deleted-messages-vault-blobstore
+
+
+
+
+
\ No newline at end of file
diff --git
a/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
new file mode 100644
index 000..33f11f6
--- /dev/null
+++
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
@@ -0,0 +1,58 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.vault;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.james.blob.api.BucketName;
+import org.apache.james.core.User;
+import org.apache.james.mailbox.model.MessageId;
+import org.reactivestreams.Publisher;
+
+public class MemoryDeletedMessageMetadataVault implements
DeletedMessageMetadataVault {
+@Override
+public Publisher store(DeletedMessageWithStorageInformation
deletedMessage) {
+throw new NotImplementedException("Not yet implemented");
+}
+
+@Override
+public Publisher removeBucket(BucketName bucketName) {
+throw new NotImplementedException("Not yet implemented");
+}
+
+@Override
+public Publisher remove(BucketName bucketName, User user, MessageId
messageId) {
+throw new NotImplementedException("Not yet implemented");
+}
+
+@Override
+public Publisher retrieveStorageInformation(User user,
MessageId messageId) {
+throw new NotImplementedException("Not yet implemented");
+}
+
+@Override
+public Publisher
listMessages(BucketName bucketName, User user) {
+throw new NotImplementedException("Not yet implemented");
+}
+
+@Override
+public Publisher listBuckets() {
+throw new NotImplementedException("Not yet implemented");
+}
+}
diff --git
a/mailbox/plugin/deleted-messages-vault-blobstore/src/main/java/org/apache/james/vault/DeletedMessageMetadataVault.java
b/mailbox/plugin/
[james-project] 01/13: JAMES-2806 add deleteBucket method in BlobStore
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7257bc83bb487a22f96449fa48ae7696d83bf322
Author: Rene Cordier
AuthorDate: Wed Jun 26 15:55:31 2019 +0700
JAMES-2806 add deleteBucket method in BlobStore
---
.../src/main/java/org/apache/james/blob/api/BlobStore.java | 2 ++
.../java/org/apache/james/blob/api/MetricableBlobStore.java| 5 +
.../org/apache/james/blob/cassandra/CassandraBlobsDAO.java | 6 ++
.../java/org/apache/james/blob/memory/MemoryBlobStore.java | 6 ++
.../apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java | 6 ++
.../main/java/org/apache/james/blob/union/UnionBlobStore.java | 7 +++
.../java/org/apache/james/blob/union/UnionBlobStoreTest.java | 10 ++
7 files changed, 42 insertions(+)
diff --git
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStore.java
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStore.java
index ed7f6f7..3302203 100644
---
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStore.java
+++
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobStore.java
@@ -36,4 +36,6 @@ public interface BlobStore {
default Mono save(BucketName bucketName, String data) {
return save(bucketName, data.getBytes(StandardCharsets.UTF_8));
}
+
+Mono deleteBucket(BucketName bucketName);
}
diff --git
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
index 3976bcd..89548d8 100644
---
a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
+++
b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/MetricableBlobStore.java
@@ -70,4 +70,9 @@ public class MetricableBlobStore implements BlobStore {
return metricFactory
.runPublishingTimerMetric(READ_TIMER_NAME, () ->
blobStoreImpl.read(bucketName, blobId));
}
+
+@Override
+public Mono deleteBucket(BucketName bucketName) {
+return blobStoreImpl.deleteBucket(bucketName);
+}
}
diff --git
a/server/blob/blob-cassandra/src/main/java/org/apache/james/blob/cassandra/CassandraBlobsDAO.java
b/server/blob/blob-cassandra/src/main/java/org/apache/james/blob/cassandra/CassandraBlobsDAO.java
index 23be4e5..38edc21 100644
---
a/server/blob/blob-cassandra/src/main/java/org/apache/james/blob/cassandra/CassandraBlobsDAO.java
+++
b/server/blob/blob-cassandra/src/main/java/org/apache/james/blob/cassandra/CassandraBlobsDAO.java
@@ -34,6 +34,7 @@ import java.util.stream.Stream;
import javax.inject.Inject;
import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.lang3.tuple.Pair;
import
org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration;
import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor;
@@ -218,4 +219,9 @@ public class CassandraBlobsDAO implements BlobStore {
return Mono.fromCallable(() -> IOUtils.toByteArray(data))
.flatMap(this::saveAsMono);
}
+
+@Override
+public Mono deleteBucket(BucketName bucketName) {
+throw new NotImplementedException("not implemented");
+}
}
diff --git
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
index 173ce78..154a9b5 100644
---
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
+++
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
@@ -28,6 +28,7 @@ import java.util.concurrent.ConcurrentHashMap;
import javax.inject.Inject;
import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.NotImplementedException;
import org.apache.james.blob.api.BlobId;
import org.apache.james.blob.api.BlobStore;
import org.apache.james.blob.api.BucketName;
@@ -78,6 +79,11 @@ public class MemoryBlobStore implements BlobStore {
return new ByteArrayInputStream(retrieveStoredValue(blobId));
}
+@Override
+public Mono deleteBucket(BucketName bucketName) {
+throw new NotImplementedException("not implemented");
+}
+
private byte[] retrieveStoredValue(BlobId blobId) {
return Optional.ofNullable(blobs.get(blobId))
.orElseThrow(() -> new ObjectStoreException("unable to find blob
with id " + blobId));
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java
index 7bbd2da..48840f8 100644
---
a/server/blob/blob-objectstorage/src/m
[james-project] 04/13: JAMES-2806 Add deleteBucket implementation for memory and its contract
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1a7e0588d24a86728dd04289c899fead15c78355
Author: Rene Cordier
AuthorDate: Mon Jul 1 10:55:21 2019 +0700
JAMES-2806 Add deleteBucket implementation for memory and its contract
---
.../james/blob/api/BucketBlobStoreContract.java| 36 ++
.../apache/james/blob/memory/MemoryBlobStore.java | 11 ---
2 files changed, 43 insertions(+), 4 deletions(-)
diff --git
a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
index 4655bb0..d52d703 100644
---
a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
+++
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
@@ -20,6 +20,7 @@
package org.apache.james.blob.api;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
@@ -39,6 +40,30 @@ public interface BucketBlobStoreContract {
BlobId.Factory blobIdFactory();
@Test
+default void deleteBucketShouldThrowWhenNullBucketName() {
+assertThatThrownBy(() -> testee().deleteBucket(null).block())
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void deleteBucketShouldDeleteExistingBucketWithItsData() {
+BlobId blobId = testee().save(CUSTOM, SHORT_BYTEARRAY).block();
+testee().deleteBucket(CUSTOM).block();
+
+assertThatThrownBy(() -> testee().read(CUSTOM, blobId))
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+default void deleteBucketShouldBeIdempotent(){
+testee().save(CUSTOM, SHORT_BYTEARRAY).block();
+testee().deleteBucket(CUSTOM).block();
+
+assertThatCode(() -> testee().deleteBucket(CUSTOM).block())
+.doesNotThrowAnyException();
+}
+
+@Test
default void saveBytesShouldThrowWhenNullBucketName() {
assertThatThrownBy(() -> testee().save(null, SHORT_BYTEARRAY).block())
.isInstanceOf(NullPointerException.class);
@@ -103,4 +128,15 @@ public interface BucketBlobStoreContract {
.operationCount(10)
.runSuccessfullyWithin(Duration.ofMinutes(1));
}
+
+@Test
+default void deleteBucketConcurrentlyShouldNotFail() throws Exception {
+testee().save(CUSTOM, SHORT_BYTEARRAY).block();
+
+ConcurrentTestRunner.builder()
+.operation(((threadNumber, step) ->
testee().deleteBucket(CUSTOM).block()))
+.threadCount(10)
+.operationCount(10)
+.runSuccessfullyWithin(Duration.ofMinutes(1));
+}
}
diff --git
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
index b7f907a..812ea3d 100644
---
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
+++
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
@@ -27,7 +27,6 @@ import java.util.Optional;
import javax.inject.Inject;
import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.NotImplementedException;
import org.apache.james.blob.api.BlobId;
import org.apache.james.blob.api.BlobStore;
import org.apache.james.blob.api.BucketName;
@@ -35,10 +34,8 @@ import org.apache.james.blob.api.ObjectStoreException;
import com.google.common.base.Preconditions;
import com.google.common.collect.HashBasedTable;
-import com.google.common.collect.ImmutableList;
import com.google.common.collect.Table;
-import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class MemoryBlobStore implements BlobStore {
@@ -92,7 +89,13 @@ public class MemoryBlobStore implements BlobStore {
@Override
public Mono deleteBucket(BucketName bucketName) {
-throw new NotImplementedException("not implemented");
+Preconditions.checkNotNull(bucketName);
+
+return Mono.fromRunnable(() -> {
+synchronized (blobs) {
+blobs.row(bucketName).clear();
+}
+});
}
private byte[] retrieveStoredValue(BucketName bucketName, BlobId blobId) {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 09/13: JAMES-2807 Implement Memory DeletedMessageMetadataVault and related contract
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 5063bfca0d28010a508fa7b132a0b6b4b199ab7d
Author: Benoit Tellier
AuthorDate: Tue Jun 25 14:14:26 2019 +0700
JAMES-2807 Implement Memory DeletedMessageMetadataVault and related contract
---
.../pom.xml| 51 +-
.../vault/MemoryDeletedMessageMetadataVault.java | 68 ++-
.../MemoryDeletedMessageMetadataVaultTest.java}| 22 ++-
.../DeletedMessageWithStorageInformation.java | 16 +-
.../vault/DeletedMessageMetadataVaultContract.java | 198 +
.../vault/DeletedMessageVaultMetadataFixture.java | 1 +
6 files changed, 331 insertions(+), 25 deletions(-)
diff --git a/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
index d5ae54e..faf54ee 100644
--- a/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
+++ b/mailbox/plugin/deleted-messages-vault-blobstore-memory/pom.xml
@@ -31,10 +31,57 @@
-org.apache.james
+${james.groupId}
+apache-james-mailbox-api
+test-jar
+test
+
+
+${james.groupId}
+
apache-james-mailbox-deleted-messages-vault
+test-jar
+test
+
+
+${james.groupId}
+
apache-james-mailbox-deleted-messages-vault-blobstore
+
+
+${james.groupId}
apache-james-mailbox-deleted-messages-vault-blobstore
+test-jar
+test
+
+
+${james.groupId}
+apache-james-mailbox-memory
+test
+
+
+${james.groupId}
+apache-james-mailbox-memory
+test
+test-jar
+
+
+com.google.guava
+guava
+
+
+org.assertj
+assertj-core
+test
+
+
+org.junit.jupiter
+junit-jupiter-engine
+test
+
+
+org.junit.platform
+junit-platform-launcher
+test
-
\ No newline at end of file
diff --git
a/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
index 33f11f6..4bafe68 100644
---
a/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
+++
b/mailbox/plugin/deleted-messages-vault-blobstore-memory/src/main/java/org/apache/james/vault/MemoryDeletedMessageMetadataVault.java
@@ -19,40 +19,94 @@
package org.apache.james.vault;
-import org.apache.commons.lang3.NotImplementedException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
import org.apache.james.blob.api.BucketName;
import org.apache.james.core.User;
import org.apache.james.mailbox.model.MessageId;
import org.reactivestreams.Publisher;
+import com.google.common.collect.HashBasedTable;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Table;
+
+import reactor.core.publisher.Flux;
+import reactor.core.publisher.Mono;
+
public class MemoryDeletedMessageMetadataVault implements
DeletedMessageMetadataVault {
+private final Table> table;
+
+public MemoryDeletedMessageMetadataVault() {
+table = HashBasedTable.create();
+}
+
@Override
public Publisher store(DeletedMessageWithStorageInformation
deletedMessage) {
-throw new NotImplementedException("Not yet implemented");
+BucketName bucketName =
deletedMessage.getStorageInformation().getBucketName();
+User owner = deletedMessage.getDeletedMessage().getOwner();
+MessageId messageId =
deletedMessage.getDeletedMessage().getMessageId();
+
+return Mono.fromRunnable(() -> {
+synchronized (table) {
+Map userVault
= userVault(bucketName, owner);
+userVault.put(messageId, deletedMessage);
+table.put(bucketName, owner, userVault);
+}
+});
}
@Override
public Publisher removeBucket(BucketName bucketName) {
-throw new NotImplementedException("Not yet implemented");
+return Mono.fromRunnable(() -> {
+synchronized (table) {
+table.row(bucketName).clear();
+}
+});
}
@Override
public Publisher remove(BucketName bucketName, User user, MessageId
messageId) {
-throw new NotImplementedException("
[james-project] 03/13: JAMES-2806 add contract for BlobStore with custom buckets
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 2672ea9c839bbaca795ebe0146bcfdaab7014a75
Author: Rene Cordier
AuthorDate: Tue Jun 25 16:49:46 2019 +0700
JAMES-2806 add contract for BlobStore with custom buckets
---
.../james/blob/api/BucketBlobStoreContract.java| 106 +
.../james/blob/memory/MemoryBlobStoreTest.java | 3 +-
2 files changed, 108 insertions(+), 1 deletion(-)
diff --git
a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
new file mode 100644
index 000..4655bb0
--- /dev/null
+++
b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BucketBlobStoreContract.java
@@ -0,0 +1,106 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.blob.api;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+
+import org.apache.james.util.concurrency.ConcurrentTestRunner;
+import org.junit.jupiter.api.Test;
+
+public interface BucketBlobStoreContract {
+String SHORT_STRING = "toto";
+byte[] SHORT_BYTEARRAY = SHORT_STRING.getBytes(StandardCharsets.UTF_8);
+BucketName CUSTOM = BucketName.of("custom");
+
+BlobStore testee();
+
+BlobId.Factory blobIdFactory();
+
+@Test
+default void saveBytesShouldThrowWhenNullBucketName() {
+assertThatThrownBy(() -> testee().save(null, SHORT_BYTEARRAY).block())
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void saveStringShouldThrowWhenNullBucketName() {
+assertThatThrownBy(() -> testee().save(null, SHORT_STRING).block())
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void saveInputStreamShouldThrowWhenNullBucketName() {
+assertThatThrownBy(() -> testee().save(null, new
ByteArrayInputStream(SHORT_BYTEARRAY)).block())
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void readShouldThrowWhenNullBucketName() {
+BlobId blobId = testee().save(BucketName.DEFAULT,
SHORT_BYTEARRAY).block();
+assertThatThrownBy(() -> testee().read(null, blobId))
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void readBytesStreamShouldThrowWhenNullBucketName() {
+BlobId blobId = testee().save(BucketName.DEFAULT,
SHORT_BYTEARRAY).block();
+assertThatThrownBy(() -> testee().readBytes(null, blobId).block())
+.isInstanceOf(NullPointerException.class);
+}
+
+@Test
+default void readStringShouldThrowWhenBucketDoesNotExist() {
+BlobId blobId = testee().save(BucketName.DEFAULT,
SHORT_BYTEARRAY).block();
+assertThatThrownBy(() -> testee().read(CUSTOM, blobId))
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+default void readBytesStreamShouldThrowWhenBucketDoesNotExist() {
+BlobId blobId = testee().save(BucketName.DEFAULT,
SHORT_BYTEARRAY).block();
+assertThatThrownBy(() -> testee().readBytes(CUSTOM, blobId).block())
+.isInstanceOf(ObjectStoreException.class);
+}
+
+@Test
+default void shouldBeAbleToSaveDataInMultipleBuckets() {
+BlobId blobIdDefault = testee().save(BucketName.DEFAULT,
SHORT_BYTEARRAY).block();
+BlobId blobIdCustom = testee().save(CUSTOM, SHORT_BYTEARRAY).block();
+
+byte[] bytesDefault = testee().readBytes(BucketName.DEFAULT,
blobIdDefault).block();
[james-project] 02/13: JAMES-2806 implement buckets with memory BlobStore
This is an automated email from the ASF dual-hosted git repository.
aduprat pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 53d3ed7317abf7f541f2fa01c1b6d0d8473da996
Author: Rene Cordier
AuthorDate: Tue Jun 25 14:50:54 2019 +0700
JAMES-2806 implement buckets with memory BlobStore
---
.../apache/james/blob/memory/MemoryBlobStore.java | 35 +++---
1 file changed, 24 insertions(+), 11 deletions(-)
diff --git
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
index 154a9b5..b7f907a 100644
---
a/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
+++
b/server/blob/blob-memory/src/main/java/org/apache/james/blob/memory/MemoryBlobStore.java
@@ -23,7 +23,6 @@ import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Optional;
-import java.util.concurrent.ConcurrentHashMap;
import javax.inject.Inject;
@@ -35,31 +34,41 @@ import org.apache.james.blob.api.BucketName;
import org.apache.james.blob.api.ObjectStoreException;
import com.google.common.base.Preconditions;
+import com.google.common.collect.HashBasedTable;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Table;
+import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class MemoryBlobStore implements BlobStore {
-private final ConcurrentHashMap blobs;
private final BlobId.Factory factory;
+private final Table blobs;
@Inject
public MemoryBlobStore(BlobId.Factory factory) {
this.factory = factory;
-blobs = new ConcurrentHashMap<>();
+blobs = HashBasedTable.create();
}
@Override
public Mono save(BucketName bucketName, byte[] data) {
+Preconditions.checkNotNull(bucketName);
Preconditions.checkNotNull(data);
-BlobId blobId = factory.forPayload(data);
-blobs.put(blobId, data);
+BlobId blobId = factory.forPayload(data);
-return Mono.just(blobId);
+return Mono.fromCallable(() -> {
+synchronized (blobs) {
+blobs.put(bucketName, blobId, data);
+return blobId;
+}
+});
}
@Override
public Mono save(BucketName bucketName, InputStream data) {
+Preconditions.checkNotNull(bucketName);
Preconditions.checkNotNull(data);
try {
byte[] bytes = IOUtils.toByteArray(data);
@@ -71,12 +80,14 @@ public class MemoryBlobStore implements BlobStore {
@Override
public Mono readBytes(BucketName bucketName, BlobId blobId) {
-return Mono.fromCallable(() -> retrieveStoredValue(blobId));
+Preconditions.checkNotNull(bucketName);
+return Mono.fromCallable(() -> retrieveStoredValue(bucketName,
blobId));
}
@Override
public InputStream read(BucketName bucketName, BlobId blobId) {
-return new ByteArrayInputStream(retrieveStoredValue(blobId));
+Preconditions.checkNotNull(bucketName);
+return new ByteArrayInputStream(retrieveStoredValue(bucketName,
blobId));
}
@Override
@@ -84,8 +95,10 @@ public class MemoryBlobStore implements BlobStore {
throw new NotImplementedException("not implemented");
}
-private byte[] retrieveStoredValue(BlobId blobId) {
-return Optional.ofNullable(blobs.get(blobId))
-.orElseThrow(() -> new ObjectStoreException("unable to find blob
with id " + blobId));
+private byte[] retrieveStoredValue(BucketName bucketName, BlobId blobId) {
+synchronized (blobs) {
+return Optional.ofNullable(blobs.get(bucketName, blobId))
+.orElseThrow(() -> new ObjectStoreException("Unable to find
blob with id " + blobId + " in bucket " + bucketName.asString()));
+}
}
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
