Re: Breaking changes regarding Thread/get implementation?
Hi Benoit, I did open a pull request following your suggestion here: https://github.com/apache/james-project/pull/547 Best regards, Quan
[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [200]'
BUILD-FAILURE: Job 'james/ApacheJames/master [master] [200]': Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/200/;>james/ApacheJames/master [master] [200]" - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3616) WebAdmin: hide Jetty version
Benoit Tellier created JAMES-3616:
-
Summary: WebAdmin: hide Jetty version
Key: JAMES-3616
URL: https://issues.apache.org/jira/browse/JAMES-3616
Project: James Server
Issue Type: Improvement
Components: webadmin
Reporter: Benoit Tellier
The JETTY version is advertized:
{code:java}
root@james-jmap-bf57d6d59-4rnfb:/# curl --head
'http://127.0.0.1:8000/users/37013...@xxx.fr'
HTTP/1.1 401 Unauthorized
Date: Thu, 22 Jul 2021 04:02:25 GMT
Access-Control-Allow-Origin: *
Access-Control-Request-Method: DELETE, GET, POST, PUT
Access-Control-Allow-Headers: Content-Type, Authorization, Accept
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(9.4.31.v20200723)
{code}
This avoids scans that could map to known CVE.
We likely should consider hiding the Server field...
Cf
https://stackoverflow.com/questions/56641783/how-to-remove-server-versionserver-jetty9-2-z-snapshot-from-spark-web-ui
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3615) S3BlobDAO emptying bucket do not take into account trucated bucket lists
Benoit Tellier created JAMES-3615: - Summary: S3BlobDAO emptying bucket do not take into account trucated bucket lists Key: JAMES-3615 URL: https://issues.apache.org/jira/browse/JAMES-3615 Project: James Server Issue Type: Bug Components: Blob Affects Versions: 3.6.0 Reporter: Benoit Tellier Fix For: 3.7.0 https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-bucket.html Using the AWS SDK Java clearly does a loop as long as the list of objects is truncated. We currently do not apply suck logic in S3BlobDAO. Obviously, deleting large buckets would fail as the bucket would not have had been well emptied before. So far only the Deleted Message Vault purge feature is impacted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [199]'
BUILD-FAILURE: Job 'james/ApacheJames/master [master] [199]': Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/199/;>james/ApacheJames/master [master] [199]" - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [198]'
BUILD-FAILURE: Job 'james/ApacheJames/master [master] [198]': Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/198/;>james/ApacheJames/master [master] [198]" - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Implement ThreadIdGuessingAlgorithm for the distributed module
On 21/07/2021 18:06, Quan tran hong wrote:
> Hi Benoit,
> Thanks for your great reviews.
>
>> Or we can just do several selects, one for each mimeMessageIds.I
> personaly likely prefer this option.
>
> This will make the upper logic layer more complicated. But I agree with
> your suggestion cause I would prefer balance and performance also.
>
>> Remember to do the delete of ThreadTable before threadtable_lookup,
> otherwise if you delete the pointer before the data you might end up in
> a case where the actual data is never deleted.
>
> Sure. Thanks for your reminder.
>
>> When/How do we call it?
> StoringThreadIdGuessingAlgorithm maybe?
Not really, this would only be called when appending messages.
Likely this would need to be hooked in DeleteMessageListener.
>
> Cheers,
> Quan
>
> Vào Th 4, 21 thg 7, 2021 vào lúc 17:03 btell...@apache.org <
> btell...@apache.org> đã viết:
>
>> Hello quan,
>>
>> Nice proposal! I think the Cassandra data model you propose is evolved
>> enough so that we start implementing it.
>>
>> Some comments inlined...
>>
>> On 21/07/2021 15:55, Quan tran hong wrote:
>>> Hi Benoit,
>>> I did have another try on this. Please have a look.
>>> CREATE threadtable
>>>
>>> CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
>>> text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
>>> mimemessageid), messageid));
>>>
>>> => Partition key: (username, mimemessageid), clustering key: messageid.
>>>
>>> [...]
>>>
>>> SELECT basesubject, threadId FROM threadtable WHERE username = 'quan'
>> AND
>>> mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
>> This looks way better to me.
>>
>> IN usage is PRIMARY KEY is discouraged as it leads to coordination
>> across partitions.
>>
>> Read more for instance in
>>
>> https://stackoverflow.com/questions/55604857/cassandra-query-performance-using-in-clause-for-one-portion-of-the-composite-pa
>>
>> Either we should move "mimeMessageId" to the clustering key (but all the
>> messages of a user, including their subject would end up in a single
>> partition, which could be quite large... for instance 1 million messages
>> x size of the mime message ids and subject could be too much, as
>> partitions are recommended to be 100MBs at most).
>>
>> Or we can just do several selects, one for each mimeMessageIds.I
>> personaly likely prefer this option.
>>> [...]
>>>
>>> CREATE threadtable_lookup for deletion purpose
>>>
>>> Supposed when we delete a message, we would need to delete that message’s
>>> thread-related data in the threadtable also. I guess we just need
>> messageId
>>> to delete that message.
>>>
>>> We would need to have another table similar to the threadtable but looked
>>> up by messageId to get the needed params for deletion query.
>>>
>>> CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
>>> mimeMessageIds set, PRIMARY KEY(messageid));
>> The set should be frozen. We will never add or remove data in it, so we
>> do not need a CRDT (Commutative Replicated Data Type).
>>
>> Forbidding adding - removing individual elements avoids lots of issues,
>> and lead to a more compact storage.
>>
>>> To ease testing, I create a table with messageId’s type is int instead.
>>>
>>> We will insert the data as same as the original table.
>>>
>>> insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
>>> values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2',
>> 'MimeMessageID3'});
>>> [...]
>>>
>>>
>>> SELECT * FROM threadtable_lookup;
>>>
>>>
>>> Now we will do a query selection by messageId to get the needed username,
>>> mimeMessageIds for original threadtable deletion.
>>>
>>> Supposed we want to delete message 4.
>>>
>>> SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid
>> = 4;
>>>
>>> Then we will use these results to do a deletion query on threadtable.
>>>
>> Remember to do the delete of ThreadTable before threadtable_lookup,
>> otherwise if you delete the pointer before the data you might end up in
>> a case where the actual data is never deleted.
>>
>> The algorithm that you propose looks good.
>>
>> When/How do we call it?
>>
>>
>> Cheers,
>>
>> Benoit
>>> The data of messageId 4 deleted.
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Quan
>>>
>>>
>>> Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
>>> btell...@apache.org> đã viết:
>>>
Hello Quan,
On 20/07/2021 17:24, Quan tran hong wrote:
> [...]
>
> SELECT threadId FROM threadtable WHERE username = 'quan' AND
>> baseSubject
=
> 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2',
>> 'MimeMessageID3')
> LIMIT 1 ALLOW FILTERING;
ALLOW FILTERING should not be used as it will result in a full scan and
is thus a performance disaster.
If you need it, this means you do not have the right table structure and
likely should rework the CREATE TABLE statement.
> => This new message should have this threadId.
> New unrelated message
Re: svn commit: r48925 - in /release/james: hupa/0.0.2/ jdkim/ jsieve/0.7/ jspf/1.0.1/ mailets/3.6.0/ mime4j/0.8.5/ mpt/0.1/ server/ server/3.6.0/
Hello all, Just a word about this, The INFRA requests us to armor our signature on the download page, I received the following message: |> Sorry, but the download page is still wrong. |||> | |||> |The links for the hupa artifacts are missing the /hupa/ subdirectory |||> |segment. > || > ||Also, I've just noticed that all the .ASC files are binary. > ||They are supposed to be ascii-armoured. > ||As it stands, the wrong content-type will be applied which may cause issues > ||with downloads. > ||Please fix for future releases, and if necessary for current releases. > ||Please check what needs to be done with INFRA ASAP.> || > ||Sebb| I did the update. See also this PR to put ourselves in conformity. This prevents us from sending anything to Apache announce mailing list. No downloadable artifact had been changed. Only signatures. Cheers, Benoit On 21/07/2021 17:58, btell...@apache.org wrote: > Author: btellier > Date: Wed Jul 21 10:58:42 2021 > New Revision: 48925 > > Log: > Follow ASF policy regarding artifact signature: use ascii-armoured instead of > binaries > > Modified: > release/james/hupa/0.0.2/hupa-0.0.2.war.asc > release/james/hupa/0.0.2/hupa-parent-0.0.2-source-release.zip.asc > release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc > release/james/jdkim/apache-jdkim-0.2-bin.zip.asc > release/james/jdkim/apache-jdkim-project-0.2-source-release.zip.asc > release/james/jsieve/0.7/apache-jsieve-0.7-all.tar.gz.asc > release/james/jsieve/0.7/apache-jsieve-core-0.7.jar.asc > release/james/jsieve/0.7/apache-jsieve-sources-0.7.zip.asc > release/james/jspf/1.0.1/apache-jspf-1.0.1-bin.zip.asc > release/james/jspf/1.0.1/apache-jspf-sources-1.0.1.zip.asc > release/james/mailets/3.6.0/apache-mailet-api-3.6.0-sources.jar.asc > release/james/mailets/3.6.0/apache-mailet-api-3.6.0.jar.asc > release/james/mailets/3.6.0/apache-mailet-base-3.6.0-sources.jar.asc > release/james/mailets/3.6.0/apache-mailet-base-3.6.0.jar.asc > release/james/mailets/3.6.0/apache-mailet-crypto-3.6.0-sources.jar.asc > release/james/mailets/3.6.0/apache-mailet-crypto-3.6.0.jar.asc > release/james/mailets/3.6.0/apache-mailet-standard-3.6.0-sources.jar.asc > release/james/mailets/3.6.0/apache-mailet-standard-3.6.0.jar.asc > release/james/mime4j/0.8.5/apache-mime4j-0.8.5-bin.tar.gz.asc > release/james/mime4j/0.8.5/apache-mime4j-0.8.5-bin.zip.asc > release/james/mime4j/0.8.5/apache-mime4j-core-0.8.5.jar.asc > release/james/mime4j/0.8.5/apache-mime4j-dom-0.8.5.jar.asc > release/james/mime4j/0.8.5/james-mime4j-sources-0.8.5.zip.asc > release/james/mpt/0.1/apache-james-mpt-0.1-src.tar.gz.asc > release/james/mpt/0.1/apache-james-mpt-0.1-src.zip.asc > release/james/server/3.6.0/james-project-3.6.0-source-release.zip.asc > release/james/server/3.6.0/james-server-app-3.6.0-app.zip.asc > release/james/server/james-2.3.2.1-src.tar.gz.asc > release/james/server/james-2.3.2.1-src.zip.asc > release/james/server/james-binary-2.3.2.1.tar.gz.asc > release/james/server/james-binary-2.3.2.1.zip.asc > > Modified: release/james/hupa/0.0.2/hupa-0.0.2.war.asc > == > Binary files - no diff available. > > Modified: release/james/hupa/0.0.2/hupa-parent-0.0.2-source-release.zip.asc > == > Binary files - no diff available. > > Modified: release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc > == > --- release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc (original) > +++ release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc Wed Jul 21 10:58:42 > 2021 > @@ -1,7 +1,11 @@ > -BEGIN PGP SIGNATURE- > -Version: GnuPG/MacGPG2 v2.0.11 (Darwin) > > -iEYEABECAAYFAk4tph0ACgkQBWrKdNRgAL+8dwCfVJjLjVAM8AZg5X/ahKaEzEM4 > -cy4An2SAMph1a4ItQ/NYNSOfpNxLCN2h > -=4pPC > +iQEyBAABCAAdFiEEFnhshRITEEQ+vk9LDhxDw/s1ts4FAmD3+18ACgkQDhxDw/s1 > +ts7/9Qf4k4ERCZpPZ00zkzNOkTR52awf7UbxDElStTQZR14s9N3tqjrUROVDtTHt > +Fk5/bqOp4LHRdEwSvU8Xni34VuYYUviUzey4zJe0YVDNWqN8nbntsxsD+TopAqo5 > +/kvkGKXDwsBPVK0QRfhTjmXroA/RykgouuTVsR7K0pkgCqaLrLrzpIzpXSVGaPQ+ > +nPXoSt2vmoglFuCxd4M+9Sqm8AahIbcddptycc5P1pQOES9be0AGhJEpUSYblO9O > +YqOjHAwuyFnx1pJK3++N5oP7PSZYkClVLG8dNbWmkrlvv4wwN7NGdFhlqL95m5ey > +26lVgM/XkZNsOCkQ7KmnX1OmzcqF > +=wsvH > -END PGP SIGNATURE- > > Modified: release/james/jdkim/apache-jdkim-0.2-bin.zip.asc > == > --- release/james/jdkim/apache-jdkim-0.2-bin.zip.asc (original) > +++ release/james/jdkim/apache-jdkim-0.2-bin.zip.asc Wed Jul 21 10:58:42 2021 > @@ -1,7 +1,11 @@ > -BEGIN PGP SIGNATURE- > -Version: GnuPG/MacGPG2 v2.0.11 (Darwin) > > -iEYEABECAAYFAk4tph0ACgkQBWrKdNRgAL/5bwCaAscqpCVk2SM8NZ25VeuzBZYV >
Re: Implement ThreadIdGuessingAlgorithm for the distributed module
Hi Benoit,
Thanks for your great reviews.
> Or we can just do several selects, one for each mimeMessageIds.I
personaly likely prefer this option.
This will make the upper logic layer more complicated. But I agree with
your suggestion cause I would prefer balance and performance also.
> Remember to do the delete of ThreadTable before threadtable_lookup,
otherwise if you delete the pointer before the data you might end up in
a case where the actual data is never deleted.
Sure. Thanks for your reminder.
> When/How do we call it?
StoringThreadIdGuessingAlgorithm maybe?
Cheers,
Quan
Vào Th 4, 21 thg 7, 2021 vào lúc 17:03 btell...@apache.org <
btell...@apache.org> đã viết:
> Hello quan,
>
> Nice proposal! I think the Cassandra data model you propose is evolved
> enough so that we start implementing it.
>
> Some comments inlined...
>
> On 21/07/2021 15:55, Quan tran hong wrote:
> > Hi Benoit,
> > I did have another try on this. Please have a look.
> > CREATE threadtable
> >
> > CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
> > text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
> > mimemessageid), messageid));
> >
> > => Partition key: (username, mimemessageid), clustering key: messageid.
> >
> > [...]
> >
> > SELECT basesubject, threadId FROM threadtable WHERE username = 'quan'
> AND
> > mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
> This looks way better to me.
>
> IN usage is PRIMARY KEY is discouraged as it leads to coordination
> across partitions.
>
> Read more for instance in
>
> https://stackoverflow.com/questions/55604857/cassandra-query-performance-using-in-clause-for-one-portion-of-the-composite-pa
>
> Either we should move "mimeMessageId" to the clustering key (but all the
> messages of a user, including their subject would end up in a single
> partition, which could be quite large... for instance 1 million messages
> x size of the mime message ids and subject could be too much, as
> partitions are recommended to be 100MBs at most).
>
> Or we can just do several selects, one for each mimeMessageIds.I
> personaly likely prefer this option.
> > [...]
> >
> > CREATE threadtable_lookup for deletion purpose
> >
> > Supposed when we delete a message, we would need to delete that message’s
> > thread-related data in the threadtable also. I guess we just need
> messageId
> > to delete that message.
> >
> > We would need to have another table similar to the threadtable but looked
> > up by messageId to get the needed params for deletion query.
> >
> > CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
> > mimeMessageIds set, PRIMARY KEY(messageid));
> The set should be frozen. We will never add or remove data in it, so we
> do not need a CRDT (Commutative Replicated Data Type).
>
> Forbidding adding - removing individual elements avoids lots of issues,
> and lead to a more compact storage.
>
> > To ease testing, I create a table with messageId’s type is int instead.
> >
> > We will insert the data as same as the original table.
> >
> > insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
> > values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2',
> 'MimeMessageID3'});
> >
> > [...]
> >
> >
> > SELECT * FROM threadtable_lookup;
> >
> >
> > Now we will do a query selection by messageId to get the needed username,
> > mimeMessageIds for original threadtable deletion.
> >
> > Supposed we want to delete message 4.
> >
> > SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid
> = 4;
> >
> >
> > Then we will use these results to do a deletion query on threadtable.
> >
> Remember to do the delete of ThreadTable before threadtable_lookup,
> otherwise if you delete the pointer before the data you might end up in
> a case where the actual data is never deleted.
>
> The algorithm that you propose looks good.
>
> When/How do we call it?
>
>
> Cheers,
>
> Benoit
> > The data of messageId 4 deleted.
> >
> >
> >
> > Best regards,
> >
> > Quan
> >
> >
> > Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
> > btell...@apache.org> đã viết:
> >
> >> Hello Quan,
> >>
> >> On 20/07/2021 17:24, Quan tran hong wrote:
> >>> [...]
> >>>
> >>> SELECT threadId FROM threadtable WHERE username = 'quan' AND
> baseSubject
> >> =
> >>> 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2',
> 'MimeMessageID3')
> >>> LIMIT 1 ALLOW FILTERING;
> >> ALLOW FILTERING should not be used as it will result in a full scan and
> >> is thus a performance disaster.
> >>
> >> If you need it, this means you do not have the right table structure and
> >> likely should rework the CREATE TABLE statement.
> >>> => This new message should have this threadId.
> >>> New unrelated message
> >>>
> >>> Assume that we do a query for a new unrelated message.
> >>>
> >>> SELECT threadId FROM threadtable WHERE username = 'quan' AND
> baseSubject
> >> =
> >>> 'unrelatedBaseSubject' AND mimeMessageId IN ('MimeMessageID2',
> >>>
svn commit: r48925 - in /release/james: hupa/0.0.2/ jdkim/ jsieve/0.7/ jspf/1.0.1/ mailets/3.6.0/ mime4j/0.8.5/ mpt/0.1/ server/ server/3.6.0/
Author: btellier Date: Wed Jul 21 10:58:42 2021 New Revision: 48925 Log: Follow ASF policy regarding artifact signature: use ascii-armoured instead of binaries Modified: release/james/hupa/0.0.2/hupa-0.0.2.war.asc release/james/hupa/0.0.2/hupa-parent-0.0.2-source-release.zip.asc release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc release/james/jdkim/apache-jdkim-0.2-bin.zip.asc release/james/jdkim/apache-jdkim-project-0.2-source-release.zip.asc release/james/jsieve/0.7/apache-jsieve-0.7-all.tar.gz.asc release/james/jsieve/0.7/apache-jsieve-core-0.7.jar.asc release/james/jsieve/0.7/apache-jsieve-sources-0.7.zip.asc release/james/jspf/1.0.1/apache-jspf-1.0.1-bin.zip.asc release/james/jspf/1.0.1/apache-jspf-sources-1.0.1.zip.asc release/james/mailets/3.6.0/apache-mailet-api-3.6.0-sources.jar.asc release/james/mailets/3.6.0/apache-mailet-api-3.6.0.jar.asc release/james/mailets/3.6.0/apache-mailet-base-3.6.0-sources.jar.asc release/james/mailets/3.6.0/apache-mailet-base-3.6.0.jar.asc release/james/mailets/3.6.0/apache-mailet-crypto-3.6.0-sources.jar.asc release/james/mailets/3.6.0/apache-mailet-crypto-3.6.0.jar.asc release/james/mailets/3.6.0/apache-mailet-standard-3.6.0-sources.jar.asc release/james/mailets/3.6.0/apache-mailet-standard-3.6.0.jar.asc release/james/mime4j/0.8.5/apache-mime4j-0.8.5-bin.tar.gz.asc release/james/mime4j/0.8.5/apache-mime4j-0.8.5-bin.zip.asc release/james/mime4j/0.8.5/apache-mime4j-core-0.8.5.jar.asc release/james/mime4j/0.8.5/apache-mime4j-dom-0.8.5.jar.asc release/james/mime4j/0.8.5/james-mime4j-sources-0.8.5.zip.asc release/james/mpt/0.1/apache-james-mpt-0.1-src.tar.gz.asc release/james/mpt/0.1/apache-james-mpt-0.1-src.zip.asc release/james/server/3.6.0/james-project-3.6.0-source-release.zip.asc release/james/server/3.6.0/james-server-app-3.6.0-app.zip.asc release/james/server/james-2.3.2.1-src.tar.gz.asc release/james/server/james-2.3.2.1-src.zip.asc release/james/server/james-binary-2.3.2.1.tar.gz.asc release/james/server/james-binary-2.3.2.1.zip.asc Modified: release/james/hupa/0.0.2/hupa-0.0.2.war.asc == Binary files - no diff available. Modified: release/james/hupa/0.0.2/hupa-parent-0.0.2-source-release.zip.asc == Binary files - no diff available. Modified: release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc == --- release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc (original) +++ release/james/jdkim/apache-jdkim-0.2-bin.tar.gz.asc Wed Jul 21 10:58:42 2021 @@ -1,7 +1,11 @@ -BEGIN PGP SIGNATURE- -Version: GnuPG/MacGPG2 v2.0.11 (Darwin) -iEYEABECAAYFAk4tph0ACgkQBWrKdNRgAL+8dwCfVJjLjVAM8AZg5X/ahKaEzEM4 -cy4An2SAMph1a4ItQ/NYNSOfpNxLCN2h -=4pPC +iQEyBAABCAAdFiEEFnhshRITEEQ+vk9LDhxDw/s1ts4FAmD3+18ACgkQDhxDw/s1 +ts7/9Qf4k4ERCZpPZ00zkzNOkTR52awf7UbxDElStTQZR14s9N3tqjrUROVDtTHt +Fk5/bqOp4LHRdEwSvU8Xni34VuYYUviUzey4zJe0YVDNWqN8nbntsxsD+TopAqo5 +/kvkGKXDwsBPVK0QRfhTjmXroA/RykgouuTVsR7K0pkgCqaLrLrzpIzpXSVGaPQ+ +nPXoSt2vmoglFuCxd4M+9Sqm8AahIbcddptycc5P1pQOES9be0AGhJEpUSYblO9O +YqOjHAwuyFnx1pJK3++N5oP7PSZYkClVLG8dNbWmkrlvv4wwN7NGdFhlqL95m5ey +26lVgM/XkZNsOCkQ7KmnX1OmzcqF +=wsvH -END PGP SIGNATURE- Modified: release/james/jdkim/apache-jdkim-0.2-bin.zip.asc == --- release/james/jdkim/apache-jdkim-0.2-bin.zip.asc (original) +++ release/james/jdkim/apache-jdkim-0.2-bin.zip.asc Wed Jul 21 10:58:42 2021 @@ -1,7 +1,11 @@ -BEGIN PGP SIGNATURE- -Version: GnuPG/MacGPG2 v2.0.11 (Darwin) -iEYEABECAAYFAk4tph0ACgkQBWrKdNRgAL/5bwCaAscqpCVk2SM8NZ25VeuzBZYV -RJkAnRyPci0gCWmLsu/aUEaocreL4gd5 -=Ux/y +iQEzBAABCAAdFiEEFnhshRITEEQ+vk9LDhxDw/s1ts4FAmD3+2kACgkQDhxDw/s1 +ts7MTwf/aKD+4/Li/u6TpgvI3e3drQhXAPVZDQ7hDHOgqDv0WdQxF/Py+qRjy7ky +XnzKTG/lqk1e2DjQTrK9hgfdoyYaUarX9e3qMGiCshSozQJBunOXQ5Z+b5mh2AZm +Xea7Wj/L/WBHJBYqE/cErVd8V8/H4dvWJuSPQGM73jcEkEe+Accfa2EFnLBKLJIw +JZ60zE84oAGfBY9NkE445+vaBIa5md2SHyVR5pHQdYPkEUqTR2NL25+xADa8+heh +csfXwXmW5aTnDXiF4JkC7MX+SvB96nT5JDCPT+cHqCDBsu0wSH1ueBTEzpuYIhn8 +R1iZKUIZrgg/WAu03KtQQPp8KontxQ== +=dxlU -END PGP SIGNATURE- Modified: release/james/jdkim/apache-jdkim-project-0.2-source-release.zip.asc == --- release/james/jdkim/apache-jdkim-project-0.2-source-release.zip.asc (original) +++ release/james/jdkim/apache-jdkim-project-0.2-source-release.zip.asc Wed Jul 21 10:58:42 2021 @@ -1,7 +1,11 @@ -BEGIN PGP SIGNATURE- -Version: GnuPG/MacGPG2 v2.0.11 (Darwin) -iEUEABECAAYFAk4tpbAACgkQBWrKdNRgAL/OOACYwgis8s5qVdFEdPsEtF86UhpO -+QCeN00bMBnsev6YMg/RzTrptXIQrxE= -=73gy +iQEzBAABCAAdFiEEFnhshRITEEQ+vk9LDhxDw/s1ts4FAmD3+3IACgkQDhxDw/s1
Re: Implement ThreadIdGuessingAlgorithm for the distributed module
Hello quan,
Nice proposal! I think the Cassandra data model you propose is evolved
enough so that we start implementing it.
Some comments inlined...
On 21/07/2021 15:55, Quan tran hong wrote:
> Hi Benoit,
> I did have another try on this. Please have a look.
> CREATE threadtable
>
> CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
> text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
> mimemessageid), messageid));
>
> => Partition key: (username, mimemessageid), clustering key: messageid.
>
> [...]
>
> SELECT basesubject, threadId FROM threadtable WHERE username = 'quan' AND
> mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
This looks way better to me.
IN usage is PRIMARY KEY is discouraged as it leads to coordination
across partitions.
Read more for instance in
https://stackoverflow.com/questions/55604857/cassandra-query-performance-using-in-clause-for-one-portion-of-the-composite-pa
Either we should move "mimeMessageId" to the clustering key (but all the
messages of a user, including their subject would end up in a single
partition, which could be quite large... for instance 1 million messages
x size of the mime message ids and subject could be too much, as
partitions are recommended to be 100MBs at most).
Or we can just do several selects, one for each mimeMessageIds.I
personaly likely prefer this option.
> [...]
>
> CREATE threadtable_lookup for deletion purpose
>
> Supposed when we delete a message, we would need to delete that message’s
> thread-related data in the threadtable also. I guess we just need messageId
> to delete that message.
>
> We would need to have another table similar to the threadtable but looked
> up by messageId to get the needed params for deletion query.
>
> CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
> mimeMessageIds set, PRIMARY KEY(messageid));
The set should be frozen. We will never add or remove data in it, so we
do not need a CRDT (Commutative Replicated Data Type).
Forbidding adding - removing individual elements avoids lots of issues,
and lead to a more compact storage.
> To ease testing, I create a table with messageId’s type is int instead.
>
> We will insert the data as same as the original table.
>
> insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
> values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2', 'MimeMessageID3'});
>
> [...]
>
>
> SELECT * FROM threadtable_lookup;
>
>
> Now we will do a query selection by messageId to get the needed username,
> mimeMessageIds for original threadtable deletion.
>
> Supposed we want to delete message 4.
>
> SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid = 4;
>
>
> Then we will use these results to do a deletion query on threadtable.
>
Remember to do the delete of ThreadTable before threadtable_lookup,
otherwise if you delete the pointer before the data you might end up in
a case where the actual data is never deleted.
The algorithm that you propose looks good.
When/How do we call it?
Cheers,
Benoit
> The data of messageId 4 deleted.
>
>
>
> Best regards,
>
> Quan
>
>
> Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
> btell...@apache.org> đã viết:
>
>> Hello Quan,
>>
>> On 20/07/2021 17:24, Quan tran hong wrote:
>>> [...]
>>>
>>> SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
>> =
>>> 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3')
>>> LIMIT 1 ALLOW FILTERING;
>> ALLOW FILTERING should not be used as it will result in a full scan and
>> is thus a performance disaster.
>>
>> If you need it, this means you do not have the right table structure and
>> likely should rework the CREATE TABLE statement.
>>> => This new message should have this threadId.
>>> New unrelated message
>>>
>>> Assume that we do a query for a new unrelated message.
>>>
>>> SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
>> =
>>> 'unrelatedBaseSubject' AND mimeMessageId IN ('MimeMessageID2',
>>> 'MimeMessageID3') LIMIT 1 ALLOW FILTERING;
>>>
>>> => This new message should have a new threadId.
>>> Insert new message data
>>>
>>> After having a threadId, we need to insert new message data into the
>> thread
>>> table.
>>>
>>> insert into ThreadTable (messageId, threadId, username, mimeMessageId,
>>> baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
>>> 'MimeMessageID2', 'baseSubject1');
>>>
>>> insert into ThreadTable (messageId, threadId, username, mimeMessageId,
>>> baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
>>> 'MimeMessageID3', 'baseSubject1');
>>> Conclusion
>>>
>>> I think this data model complies with the needed request for the guessing
>>> algorithm problem, but it looks like still maybe there is room for
>>> improvement.
>> What Cassandra request do we use to delete the data in there?
>>
>>>
>>> Best Regards,
>>>
>>> Quan
>>>
>>>
>>>
>>>
>>>
>>> Vào Th 2, 19 thg 7, 2021
[ADR] JAMES-3544 cleanup of JMAP uploads
Hello James devs, I would like to discuss with you the design of JMAP uploads, the way to restructure them and eventually clean them up. This separation between upload entity and attachment entity enable a rework of the attachment api and Cassandra data-model leading to simplifications. Here is the link to the ADR: https://github.com/apache/james-project/pull/544 Cheers, Benoit - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Implement ThreadIdGuessingAlgorithm for the distributed module
Hi Benoit,
I did have another try on this. Please have a look.
CREATE threadtable
CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
mimemessageid), messageid));
=> Partition key: (username, mimemessageid), clustering key: messageid.
To ease testing, I create a table with messageId and threadId type is int
instead.
Insert data
I will add:
-
2 related message and 1 unrelated message for user ‘quan’
-
1 message for user ‘benoit’ (which seem related to some messages of user
‘quan’)
// insert message1 data for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID1', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID2', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (1, 1, 'quan', 'MimeMessageID3', 'baseSubject1');
// insert message2 data (related to message1) for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (2, 1, 'quan', 'MimeMessageID1', 'baseSubject1');
// insert message3 data (not related to any message) for ‘quan’
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (3, 3, 'quan', 'MimeMessageID4', 'baseSubject2');
// insert message4 data (related to message1 but for ‘benoit’)
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (4, 4, 'benoit', 'MimeMessageID5', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (4, 4, 'benoit', 'MimeMessageID1', 'baseSubject1');
Select all data
The query for new messages
Because the subject can be null and Cassandra does not support selection by
null, so I will leave the base subject equal condition to the upper logic
layer.
Add a new related message
For example, there is a new message coming for user ‘quan’, with some
header fields:
-
SET MimeMessageIds (after combine values): {‘MimeMessageID2’,
‘MimeMessageID3’}
-
Base subject line (after stripping): “baseSubject1”
This message is supposed to be related to 2 other messages of ‘quan’.
We need to query one row related to this new message (if there is).
SELECT basesubject, threadId FROM threadtable WHERE username = 'quan' AND
mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
We will get these results and handle the rest subject condition at upper
level.
=> After that, this new message should have old threadId 1.
Add a new unrelated message
Supposed that we would add a message with unrelated mimemessage ids
SELECT basesubject, threadId FROM threadtable WHERE username = 'quan' AND
mimeMessageId IN (‘unrelatedMimeMessageID’);
As we can see, the identical mimeMessageID is not satisfied. So we will
immediately give this message a new threadId.
Insert new message data
After having a threadId, we need to insert new message data into the thread
table.
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (5, 1, 'quan', 'MimeMessageID2', 'baseSubject1');
insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (5, 1, 'quan', 'MimeMessageID3', 'baseSubject1');
CREATE threadtable_lookup for deletion purpose
Supposed when we delete a message, we would need to delete that message’s
thread-related data in the threadtable also. I guess we just need messageId
to delete that message.
We would need to have another table similar to the threadtable but looked
up by messageId to get the needed params for deletion query.
CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
mimeMessageIds set, PRIMARY KEY(messageid));
To ease testing, I create a table with messageId’s type is int instead.
We will insert the data as same as the original table.
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2', 'MimeMessageID3'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (2, 'quan', {'MimeMessageID1'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (3, 'quan', {'MimeMessageID4'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (4, 'benoit', {'MimeMessageID5', 'MimeMessageID1'});
insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
values (5, 'quan', {'MimeMessageID2', 'MimeMessageID3'});
SELECT * FROM threadtable_lookup;
Now we will do a query selection by messageId to get the needed username,
mimeMessageIds for original threadtable deletion.
Supposed we want to delete message 4.
SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid = 4;
Then we will use these results to do a
[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [197]'
BUILD-FAILURE: Job 'james/ApacheJames/master [master] [197]': Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/197/;>james/ApacheJames/master [master] [197]" - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[BUILD-FAILURE]: Job 'james/ApacheJames/master [master] [196]'
BUILD-FAILURE: Job 'james/ApacheJames/master [master] [196]': Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/196/;>james/ApacheJames/master [master] [196]" - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
