Re[2]: Tests of mail list.
NJB> Eugene,
NJB> Re-reading your note, it sounds like you are seeing something other than
NJB> what I initially thought. You are describing delivery time for a single
NJB> message to the list?
Yes, one letter.
NJB> In that case, membership list handling wouldn't be the
NJB> problem, since it only occurs once for the message. In a very busy mailing
NJB> list server, it would be more of an issue because of the large numbers of
NJB> objects, which would create work for the garbage collector.
NJB> Instead, you appear to be seeing performance in LocalDelivery related to
NJB> saving the message for each user. Although there isn't anything that we can
NJB> do in James v2 to avoid saving a copy of the message for each user, there is
NJB> a change in the v2.2.0a1 test builld that would definitely impact
NJB> performance in that area.
NJB> It would help to know precisely where James is spending its time during your
NJB> test(s).
OK. I downloaded this version and retested for McKoi1.0 and MySQL 3.23.32
The results are attached
The legend of results is
*5000 users* - number of users in mail list
*Peak memory usage 283 460 Kb* - memory is occurred by JVM
*{07:56:52} 0 / - empty*
*{07:57:02} 0 / root*
*{08:04:24} 2731 / transport*
There
{07:56:52} - time. The 10 sec period for checking is used.
0, 2731 - number of rows in inbox table. Result of SQL - "SELECT
count(message_name) FROM inbox";
the database was being cleared before each test. it's number of
delivered letters in this time.
root - the state of spooled letter - "SELECT message_state FROM spool"
empty - no spooled message.
The McKoi1.0 with James 2.2.0a1 has in whole the same result.
The MySQL is faster than McKoi.
BUT
With MySQL It is impossible to check for list more 2500 users. The James
throws a exception "javax.mail.internet.ParseException: Out of data at position
5" for mail list with 3500, 5000, and more users.
javax.mail.internet.ParseException: Out of data at position 5
at org.apache.mailet.MailAddress.(MailAddress.java:177)
at
org.apache.james.mailrepository.JDBCMailRepository.retrieve(JDBCMailRepository.java:
709)
at org.apache.james.transport.JamesSpoolManager.run(JamesSpoolManager.java:351)
at
org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:89)
at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:125)
java.lang.RuntimeException: Exception while retrieving mail: Out of data at position 5
at
org.apache.james.mailrepository.JDBCMailRepository.retrieve(JDBCMailRepository.java:
732)
at org.apache.james.transport.JamesSpoolManager.run(JamesSpoolManager.java:351)
at
org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:89)
at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:125)
NJB> There are some people using James for large mailing lists. Hopefully one of
NJB> them will provide some input on how they are doing that right now.
NJB> --- Noel
NJB> -
NJB> To unsubscribe, e-mail: [EMAIL PROTECTED]
NJB> For additional commands, e-mail: [EMAIL PROTECTED]
Best regards, Eugene mailto:[EMAIL PROTECTED]
1000 users
peak memory usage - 217 332 Kb
{06:12:27} 0 / - empty
{06:12:37} 0 / transport
{06:12:48} 64 / transport
{06:12:58} 440 / transport
{06:13:08} 875 / transport
{06:13:18} 1000 / - empty
{06:13:28} 1000 / - empty
{06:13:38} 1000 / - empty
2500 users
peak memory usage - 282548 Kb
{06:30:21} 0 / - empty
{06:30:31} 0 / transport
{06:30:41} 0 / transport
{06:30:51} 0 / transport
{06:31:01} 10 / transport
{06:31:11} 187 / transport
{06:31:21} 369 / transport
{06:31:31} 555 / transport
{06:31:41} 742 / transport
{06:31:51} 934 / transport
{06:32:01} 1127 / transport
{06:32:11} 1320 / transport
{06:32:21} 1494 / transport
{06:32:31} 1685 / transport
{06:32:41} 1877 / transport
{06:32:51} 2072 / transport
{06:33:01} 2267 / transport
{06:33:11} 2466 / transport
{06:33:21} 2500 / - empty
5000 users
peak memory usage - 34 464 Kb
{06:58:33} 0 / - empty
{06:58:43} 0 / - empty
{06:58:53} 0 / root
{06:59:03} 0 / root
{06:59:13} 0 / root
{06:59:23} 0 / root
{06:59:33} 0 / root
{06:59:43} 0 / root
{06:59:53} 0 / root
{07:00:03} 0 / root
{07:00:13} 0 / root
{07:00:23} 0 / root
{07:00:33} 0 / root
{07:00:43} 0 / root
{07:00:53} 0 / root
{07:01:03} 0 / root
{07:01:13} 0 / root
{07:01:23} 0 / root
{07:01:33} 0 / root
{07:01:43} 0 / root
{07:01:53} 0 / root
{07:02:03} 0 / root
{07:02:13} 0 / root
{07:02:23} 0 / root
{07:02:33} 0 / root
{07:02:43} 0 / root
{07:02:53} 0 / root
{07:03:03} 0 / root
{07:03:13} 0 / root
The exception that James throws:
javax.mail.in
Re[2]: Tests of mail list.
NJB> Eugene, NJB> If you are testing, try the 2.2.0a1 test build. It has some performance NJB> improvements that may impact your application. OK. I downloaded this version and am trying. >> the Database repository is used. The database is McKoi 0.94h. NJB> Are you aware of a problem with very slow inserts in McKoi? NJB> see: http://www.mckoi.com/database/FAQ.html#9_3 NJB> Also, McKoi v1.0 apparently improved the page cache, and fixed memory leaks. NJB> See their change log. Yes, the McKoi is slower MySQL. But I think it's no cause of this behavior. Therefore I made tests for MySQL also. The results(strange for MySQL) in next letter. >> - the JVM is started with settings -Xms256m -Xmx1024m. In default >> (64Mb) the Out of Memory occurs for 7500(and more) users in mail list NJB> If you are doing memory profiling, please let us know the types and origin NJB> of the objects taking up the heap. If they are LinkedList.Entry objects, NJB> they will confirm my suspicions. I'm checking the whole memory that JVM demands. I'll try retest with memory profiling >> - in settings - 2 NJB> This should not matter for local delivery. >> 1) Why does the speed of delivery decrease after 2500 users? >> 2) Why does James use such big memory for sending letters? NJB> There is a known issue with the list() methods on the repositories. NJB> Partially it is a mismatch of Avalon repository methods returning an NJB> Iterator and JavaMail methods wanting a Collection. So the repository clones NJB> its Collection and returns an Iterator, but then the caller uses the NJB> Iterator to populate a new Collection. NJB> And sometimes there is just some code that needs to be better optimized. In NJB> the case of AbstractJdbcUsersRepository, listAllUsers() makes a LinkedList NJB> and returns an Iterator. Then listUserNames() makes a LinkedList, which it NJB> populates from the Iterator returned by listAllUsers(). Then list() returns NJB> an Iterator on listUserNames(). Finally, getMembers() creates a Vector from NJB> the Iterator it received from list(). This happens for every message. NJB> LinkedLists are terrible inefficient for space, and the cloning operations NJB> are terribly slow. I've done some optimizing inside of the mail NJB> repositories, but no one has done any inside of the user repositories. I'll NJB> see if I can make a quick fix, and we can see if it helps. OK. Clearly. >> 3) Also I see that in table spool in recipients field the ALL >>emails of mail list are placed. May be better for delivery to cut >>this to a little lists (up to 1000 or 2000 users) and send them NJB> separately? NJB> Unclear. The other changes are more likely to improve performance. I mean that in spool the letter is placed. This letter contains all emails of list. May be better to break this for some letters that will contain part emails of list. For example one letter was spooled with 'recipients' like testUser1TestUser2...testUsertestUser1 I mean next first spooled letter is spooled with 'recipients' like testUser1TestUser2...testUser1000 second spooled letter - testUser1001TestUser2...testUser2000 third spooled letter testUser2001TestUser2...testUser3000 ... And up to 10th spooled letter In other words Can process the huge mail list in parts? NJB> --- Noel NJB> - NJB> To unsubscribe, e-mail: [EMAIL PROTECTED] NJB> For additional commands, e-mail: [EMAIL PROTECTED] Best regards, Eugene mailto:[EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tests of mail list.
Eugene, LOL It is never what one thinks it might be ... I've been able to reproduce your results, and the problems you encountered were in none of the places I had expected. I didn't get a single one right. :-) At the moment, it appears that the biggest part of the delay comes from: AbstractJdbcUsersRepository.getUserByName(String, boolean) For 2500 users, it takes ~30ms per call, two calls per call to an existing mailbox. The first time that a mailbox is used, there is a 40ms in overhead setting it up. Actually, those figures are for a system that has already been jitted. Earlier on those figures can be 2x - 3x higher. Those figures total to about 100ms per message, matching your 10 messages per second value. When I changed from 2500 users to 5000, the cost of getUserByName went to ~50ms. When I went to 10,000 users, the cost of getUserByName went to ~80ms, also tracking your experience. So that is something to look at, since the code within James should not be sensitive to the size of the repository. AbstractJdbcUsersRepository.getUserByName is too slow considering the frequency with which it is called. A user cache based upon a ReferenceMap might help. The drawback is that changes made directly via SQL would not be picked up until the record is re-read, either due to gc or restart. The memory problem doesn't appear to be what I had expected, either. I'm running the test again with -Xrunhprof enabled, and will look in the morning to see what shows up. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/xdocs provided_mailets_2_1.xml
noel2003/05/29 13:40:19 Modified:src/xdocs provided_mailets_2_1.xml Log: document multiple gateway tags Revision ChangesPath 1.10 +12 -6 jakarta-james/src/xdocs/provided_mailets_2_1.xml Index: provided_mailets_2_1.xml === RCS file: /home/cvs/jakarta-james/src/xdocs/provided_mailets_2_1.xml,v retrieving revision 1.9 retrieving revision 1.10 diff -u -r1.9 -r1.10 --- provided_mailets_2_1.xml 14 May 2003 15:54:23 - 1.9 +++ provided_mailets_2_1.xml 29 May 2003 20:40:18 - 1.10 @@ -170,12 +170,18 @@ by this Mailet. Defaults to 60 seconds. deliveryThreads (optional) - The number of threads this Mailet will use to generate SMTP connections. -gateway (optional) - The host name of the SMTP server to be used as a gateway for -this server. If this value is set, then all messages will be delivered to the gateway server, -regardless of recipient address. If this value is unset, delivery will occur to SMTP servers resolved -by MX lookup. -gatewayPort (optional) - The port number of the SMTP server to be used as a gateway for -this server. This value is required if gateway is set. +gateway (optional) - The host name of the SMTP server +to be used as a gateway for this server. If this value is set, then all +messages will be delivered to the gateway server, regardless of recipient +address. To specify more than one gateway server, add multiple gateway tags, +each containing one value. If more than one server is specified, they will be +tried in order until one is successful. In addition the port may be specified +for each gateway in the format: . If this +value is unset, delivery will occur to SMTP servers resolved by MX lookup. +gatewayPort (optional) - The default port number of the +SMTP server to be used as a gateway for this server. This value will be +employed when a gateway is set and the gateway value does not specify +a port as described above. bind (optional) - If present, this value is a string describing the local IP address to which the mailet should be bound while delivering emails. If the tag is absent then the service will bind to the - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/xdocs provided_mailets_2_1.xml
noel2003/05/29 13:39:44 Modified:src/xdocs Tag: branch_2_1_fcs provided_mailets_2_1.xml Log: document multiple gateway tags Revision ChangesPath No revision No revision 1.5.4.5 +12 -6 jakarta-james/src/xdocs/provided_mailets_2_1.xml Index: provided_mailets_2_1.xml === RCS file: /home/cvs/jakarta-james/src/xdocs/provided_mailets_2_1.xml,v retrieving revision 1.5.4.4 retrieving revision 1.5.4.5 diff -u -r1.5.4.4 -r1.5.4.5 --- provided_mailets_2_1.xml 14 May 2003 15:55:12 - 1.5.4.4 +++ provided_mailets_2_1.xml 29 May 2003 20:39:44 - 1.5.4.5 @@ -170,12 +170,18 @@ by this Mailet. Defaults to 60 seconds. deliveryThreads (optional) - The number of threads this Mailet will use to generate SMTP connections. -gateway (optional) - The host name of the SMTP server to be used as a gateway for -this server. If this value is set, then all messages will be delivered to the gateway server, -regardless of recipient address. If this value is unset, delivery will occur to SMTP servers resolved -by MX lookup. -gatewayPort (optional) - The port number of the SMTP server to be used as a gateway for -this server. This value is required if gateway is set. +gateway (optional) - The host name of the SMTP server +to be used as a gateway for this server. If this value is set, then all +messages will be delivered to the gateway server, regardless of recipient +address. To specify more than one gateway server, add multiple gateway tags, +each containing one value. If more than one server is specified, they will be +tried in order until one is successful. In addition the port may be specified +for each gateway in the format: . If this +value is unset, delivery will occur to SMTP servers resolved by MX lookup. +gatewayPort (optional) - The default port number of the +SMTP server to be used as a gateway for this server. This value will be +employed when a gateway is set and the gateway value does not specify +a port as described above. bind (optional) - If present, this value is a string describing the local IP address to which the mailet should be bound while delivering emails. If the tag is absent then the service will bind to the - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/java/org/apache/james/transport/mailets GenericListserv.java
noel2003/05/29 13:39:02
Modified:src/java/org/apache/james/transport/mailets
GenericListserv.java
Log:
Remove redundant Collection copy
Revision ChangesPath
1.19 +1 -2
jakarta-james/src/java/org/apache/james/transport/mailets/GenericListserv.java
Index: GenericListserv.java
===
RCS file:
/home/cvs/jakarta-james/src/java/org/apache/james/transport/mailets/GenericListserv.java,v
retrieving revision 1.18
retrieving revision 1.19
diff -u -r1.18 -r1.19
--- GenericListserv.java 21 May 2003 04:30:29 - 1.18
+++ GenericListserv.java 29 May 2003 20:39:01 - 1.19
@@ -287,8 +287,7 @@
*/
public final void service(Mail mail) throws MessagingException {
try {
-Collection members = new Vector();
-members.addAll(getMembers());
+Collection members = getMembers();
//Check for members only flag
if (isMembersOnly() && !members.contains(mail.getSender())) {
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/java/org/apache/james/transport/mailets GenericListserv.java
noel2003/05/29 13:31:17
Modified:src/java/org/apache/james/transport/mailets Tag:
branch_2_1_fcs GenericListserv.java
Log:
Remove redundant Collection copy
Revision ChangesPath
No revision
No revision
1.12.4.5 +1 -2
jakarta-james/src/java/org/apache/james/transport/mailets/GenericListserv.java
Index: GenericListserv.java
===
RCS file:
/home/cvs/jakarta-james/src/java/org/apache/james/transport/mailets/GenericListserv.java,v
retrieving revision 1.12.4.4
retrieving revision 1.12.4.5
diff -u -r1.12.4.4 -r1.12.4.5
--- GenericListserv.java 21 May 2003 04:30:48 - 1.12.4.4
+++ GenericListserv.java 29 May 2003 20:31:17 - 1.12.4.5
@@ -286,8 +286,7 @@
*/
public final void service(Mail mail) throws MessagingException {
try {
-Collection members = new Vector();
-members.addAll(getMembers());
+Collection members = getMembers();
//Check for members only flag
if (isMembersOnly() && !members.contains(mail.getSender())) {
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/java/org/apache/james/userrepository AbstractJdbcUsersRepository.java
noel2003/05/29 13:28:47
Modified:src/java/org/apache/james/userrepository
AbstractJdbcUsersRepository.java
Log:
Improve performance of listUserNames()
Revision ChangesPath
1.22 +30 -6
jakarta-james/src/java/org/apache/james/userrepository/AbstractJdbcUsersRepository.java
Index: AbstractJdbcUsersRepository.java
===
RCS file:
/home/cvs/jakarta-james/src/java/org/apache/james/userrepository/AbstractJdbcUsersRepository.java,v
retrieving revision 1.21
retrieving revision 1.22
diff -u -r1.21 -r1.22
--- AbstractJdbcUsersRepository.java 8 Mar 2003 21:14:11 - 1.21
+++ AbstractJdbcUsersRepository.java 29 May 2003 20:28:47 - 1.22
@@ -64,9 +64,10 @@
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
+import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
-import java.util.LinkedList;
+import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -211,7 +212,7 @@
destUrl += "/";
}
// Split on "/", starting after "db://"
-List urlParams = new LinkedList();
+List urlParams = new ArrayList();
int start = 5;
int end = destUrl.indexOf('/', start);
while ( end > -1 ) {
@@ -392,6 +393,21 @@
}
}
+/**
+ * Produces the complete list of User names, with correct case.
+ * @return a List of Strings representing
+ * user names.
+ */
+protected List listUserNames() {
+Collection users = getAllUsers();
+List userNames = new ArrayList(users.size());
+for (Iterator it = users.iterator(); it.hasNext(); ) {
+userNames.add(((User)it.next()).getUserName());
+}
+users.clear();
+return userNames;
+}
+
//
// Superclass methods - overridden from AbstractUsersRepository
//
@@ -400,15 +416,23 @@
* @return an Iterator of JamesUsers.
*/
protected Iterator listAllUsers() {
-List userList = new LinkedList(); // Build the users into this list.
+return getAllUsers().iterator();
+}
+
+/**
+ * Returns a list populated with all of the Users in the repository.
+ * @return a Collection of JamesUsers.
+ */
+private Collection getAllUsers() {
+List userList = new ArrayList(); // Build the users into this list.
Connection conn = openConnection();
PreparedStatement getUsersStatement = null;
ResultSet rsUsers = null;
try {
// Get a ResultSet containing all users.
-getUsersStatement =
-conn.prepareStatement(m_getUsersSql);
+getUsersStatement =
+ conn.prepareStatement(m_getUsersSql);
rsUsers = getUsersStatement.executeQuery();
// Loop through and build a User for every row.
@@ -427,7 +451,7 @@
theJDBCUtil.closeJDBCConnection(conn);
}
-return userList.iterator();
+return userList;
}
/**
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
cvs commit: jakarta-james/src/java/org/apache/james/userrepository AbstractJdbcUsersRepository.java
noel2003/05/29 13:22:47
Modified:src/java/org/apache/james/userrepository Tag: branch_2_1_fcs
AbstractJdbcUsersRepository.java
Log:
Improve performance of listUserNames()
Revision ChangesPath
No revision
No revision
1.14.4.3 +26 -3
jakarta-james/src/java/org/apache/james/userrepository/AbstractJdbcUsersRepository.java
Index: AbstractJdbcUsersRepository.java
===
RCS file:
/home/cvs/jakarta-james/src/java/org/apache/james/userrepository/AbstractJdbcUsersRepository.java,v
retrieving revision 1.14.4.2
retrieving revision 1.14.4.3
diff -u -r1.14.4.2 -r1.14.4.3
--- AbstractJdbcUsersRepository.java 8 Mar 2003 21:54:10 - 1.14.4.2
+++ AbstractJdbcUsersRepository.java 29 May 2003 20:22:46 - 1.14.4.3
@@ -204,7 +204,7 @@
destUrl += "/";
}
// Split on "/", starting after "db://"
-List urlParams = new LinkedList();
+List urlParams = new ArrayList();
int start = 5;
int end = destUrl.indexOf('/', start);
while ( end > -1 ) {
@@ -385,6 +385,21 @@
}
}
+/**
+ * Produces the complete list of User names, with correct case.
+ * @return a List of Strings representing
+ * user names.
+ */
+protected List listUserNames() {
+Collection users = getAllUsers();
+List userNames = new ArrayList(users.size());
+for (Iterator it = users.iterator(); it.hasNext(); ) {
+userNames.add(((User)it.next()).getUserName());
+}
+users.clear();
+return userNames;
+}
+
//
// Superclass methods - overridden from AbstractUsersRepository
//
@@ -393,7 +408,15 @@
* @return an Iterator of JamesUsers.
*/
protected Iterator listAllUsers() {
-List userList = new LinkedList(); // Build the users into this list.
+return getAllUsers().iterator();
+}
+
+/**
+ * Returns a list populated with all of the Users in the repository.
+ * @return a Collection of JamesUsers.
+ */
+private Collection getAllUsers() {
+List userList = new ArrayList(); // Build the users into this list.
Connection conn = openConnection();
PreparedStatement getUsersStatement = null;
@@ -420,7 +443,7 @@
theJDBCUtil.closeJDBCConnection(conn);
}
-return userList.iterator();
+return userList;
}
/**
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: New Redirect & sons becoming available
> I see the trick: the various AddressMarker.XXX are not final,
> so you could code the static {try {...}...} block without
> getting the compile error!
Right. :-) But it is private, and used to initialize the final members of
the visible class.
> > By the way, should the SpecialAddress markers be treated
> > as case insensitive?
> What do you mean? Right now in the Redirect code only the equals
> method is used, which does an equalsIgnoreCase on the user and
> host parts.
I was refering to the use of String.compareTo, e.g.,
String addressList = getInitParameter("recipients");
if(addressList.compareTo("sender") == 0) {
newRecipients.add(SpecialAddress.SENDER);
}
If you look at getTypeCode, you'll see how we convert to lower case before
comparing. Perhaps it would be good to have a getSpecialAddress that is
similar to getTypeCode, and returns null if the address isn't one of the
special ones.
Sounds like you're doing great work. :-) Looking forward to seeing it.
--- Noel
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tests of mail list.
Eugene, Re-reading your note, it sounds like you are seeing something other than what I initially thought. You are describing delivery time for a single message to the list? In that case, membership list handling wouldn't be the problem, since it only occurs once for the message. In a very busy mailing list server, it would be more of an issue because of the large numbers of objects, which would create work for the garbage collector. Instead, you appear to be seeing performance in LocalDelivery related to saving the message for each user. Although there isn't anything that we can do in James v2 to avoid saving a copy of the message for each user, there is a change in the v2.2.0a1 test builld that would definitely impact performance in that area. It would help to know precisely where James is spending its time during your test(s). There are some people using James for large mailing lists. Hopefully one of them will provide some input on how they are doing that right now. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: New Redirect & sons becoming available
Noel,
> > 1) I didn't find another way to overcome [the] ParseException.
>
> I have not yet tested the code, but I did compile Redirect.java and
> Bounce.java. This compiles against the CVS +
> MailetContext.sendMail(Mail).
Thanks for the help. I see the trick: the various AddressMarker.XXX are not final, so
you could code the static {try {...}...} block without getting the compile error!
>
> > 2) [We] have to remember that NULL will have a different meaning than
> null.
>
> Well, it is only visible as SpecialAddress.NULL, and RFC 2821 refers to it
> as the null reverse-path, which specific instructions (section 6.1):
>
> If there is a delivery failure after acceptance of a message, the
> receiver-SMTP MUST formulate and mail a notification message. This
> notification MUST be sent using a null ("<>") reverse path in the
> envelope. The recipient of this notification MUST be the address
> from the envelope return path (or the Return-Path: line). However,
> if this address is null ("<>"), the receiver-SMTP MUST NOT send a
> notification.
>
> By the way, should the SpecialAddress markers be treated as case
> insensitive?
What do you mean? Right now in the Redirect code only the equals method is used, which
does an equalsIgnoreCase on the user and host parts. Perhaps having it all lower case
will reduce the risk of potential problems arising from those "unexistent" addresses,
if used in some wrong places. I'll change it.
>
> > 3) I'm exploring the convenience of having an AbstractRedirect class,
> >extended by Redirect and the other mailets
>
> OK.
I will in the next days complete Bounce (the getTo methods are wrong), test Bounce and
NotifyPostmaster, then introduce:
AbstractRedirect
AbstractNotify extends AbstractRedirect
Forward extends AbstractRedirect
and have:
Redirect extends AbstractRedirect
Bounce extends AbstractRedirect
NotifySender extends AbstractNotify
NotifyPostmaster extends AbstractNotify
This should get us to a less redundant code; I'll look also for small shared pieces of
code that can be put inside some convenience methods.
Vincenzo
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Mailing List getMembers()
Sounds good. -Original Message- From: Noel J. Bergman [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 10:10 AM To: James-Dev Mailing List Subject: Mailing List getMembers() Mark, Would you please look at changing Vector to ArrayList, and also look at implementing a cache using a SoftReference? Use the SoftReference to point to the current member Collection. Clear it when subscribing a new member. To support manual database changes, we can add an administrative command to also clear the SoftReference. This would cut down on user repository operations, and associated heap churning, in an active mailing list server. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Mailing List getMembers()
Mark, Would you please look at changing Vector to ArrayList, and also look at implementing a cache using a SoftReference? Use the SoftReference to point to the current member Collection. Clear it when subscribing a new member. To support manual database changes, we can add an administrative command to also clear the SoftReference. This would cut down on user repository operations, and associated heap churning, in an active mailing list server. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Shutting down James
Noel, I didn't have to 'do' anything to James to get the JMX stop/start mechanism working, all that is provided by the Phoenix SystemManager / JMX implementation. What I (thought I) said was 'we made some minor changes to the dispose() methods'. This is because some of the components were shutting down untidily and throwing a bunch of exceptions. Mainly trying to use a context log object after that object had already been removed. I would like to improve the JMX monitoring and management of James, but don't hold your breath as I have a lot of things on my plate right now! Cheers Steve > -Original Message- > From: Noel J. Bergman [mailto:[EMAIL PROTECTED] > Sent: Thursday, May 29, 2003 9:04 AM > To: James Developers List > Subject: RE: Shutting down James > > > > I am working on JMX stuff right now. My objective is to provide > > fine-grain redeployment of an appliance inside a container. This > > means that all components would be redeployable (without changing > > existing component code). > > Great. :-) > > > this will not bne available tommorow > > Understood. We haven't been rushing at JMX. When the > support is there from Avalon, we'll want to take advantage of > it. If Steve has some stuff already working for James, > that'll be good to look at, too. > > --- Noel > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: XMLResources and demo mailet
Mark, Would you like to revise the list manager to use XMLResources, and submit? I'll go ahead and commit XMLResources. Go ahead and remove CDATA except from the footer_html and admincommands text resource elements. Both of them need it, since they have embedded HTML. If you get it wrong you'll know from the parsing error at startup. --- Noel -Original Message- From: Mark Imel [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 12:22 To: James Developers List Subject: RE: XMLResources and demo mailet Noel, there's a lot a like about this approach. You have the convience class 'XMLResources' that hides a lot of the heavy lifting, and makes the client's task easy. As i noted before in the ListManager proposal, anything to get the custom resources out of the class path, and into an adminstratively editable place was crucial to making the listManager useful. Also, i really like the 'overloading' concept of the 'for' attribute in regards to the matcher. That provides a nice way to customize the resources on a per group basis. The only thing that i don't like, is mixing in actual content (the CDATA) into the xml. It's not that i object to the location of the CDATA, it just makes for some ugly looking xml. It's really just a minor nit, and at this point, the positives far outweigh the negatives. Nice work. How do you want to proceed with this improvement? -Original Message- From: Noel J. Bergman [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 28, 2003 8:17 PM To: James Developers List Subject: RE: XMLResources and demo mailet > I haven't gotten the responses.properties table in there They are present in the revision attached. My expectation is that the groups would be used to organize messages for each command, and the "for" matching would be used to allow a particular mailing list to replace one or more message resources. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tests of mail list.
Eugene, If you are testing, try the 2.2.0a1 test build. It has some performance improvements that may impact your application. > the Database repository is used. The database is McKoi 0.94h. Are you aware of a problem with very slow inserts in McKoi? see: http://www.mckoi.com/database/FAQ.html#9_3 Also, McKoi v1.0 apparently improved the page cache, and fixed memory leaks. See their change log. > - the JVM is started with settings -Xms256m -Xmx1024m. In default > (64Mb) the Out of Memory occurs for 7500(and more) users in mail list If you are doing memory profiling, please let us know the types and origin of the objects taking up the heap. If they are LinkedList.Entry objects, they will confirm my suspicions. > - in settings - 2 This should not matter for local delivery. > 1) Why does the speed of delivery decrease after 2500 users? > 2) Why does James use such big memory for sending letters? There is a known issue with the list() methods on the repositories. Partially it is a mismatch of Avalon repository methods returning an Iterator and JavaMail methods wanting a Collection. So the repository clones its Collection and returns an Iterator, but then the caller uses the Iterator to populate a new Collection. And sometimes there is just some code that needs to be better optimized. In the case of AbstractJdbcUsersRepository, listAllUsers() makes a LinkedList and returns an Iterator. Then listUserNames() makes a LinkedList, which it populates from the Iterator returned by listAllUsers(). Then list() returns an Iterator on listUserNames(). Finally, getMembers() creates a Vector from the Iterator it received from list(). This happens for every message. LinkedLists are terrible inefficient for space, and the cloning operations are terribly slow. I've done some optimizing inside of the mail repositories, but no one has done any inside of the user repositories. I'll see if I can make a quick fix, and we can see if it helps. > 3) Also I see that in table spool in recipients field the ALL >emails of mail list are placed. May be better for delivery to cut >this to a little lists (up to 1000 or 2000 users) and send them separately? Unclear. The other changes are more likely to improve performance. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: XMLResources and demo mailet
Noel, there's a lot a like about this approach. You have the convience class 'XMLResources' that hides a lot of the heavy lifting, and makes the client's task easy. As i noted before in the ListManager proposal, anything to get the custom resources out of the class path, and into an adminstratively editable place was crucial to making the listManager useful. Also, i really like the 'overloading' concept of the 'for' attribute in regards to the matcher. That provides a nice way to customize the resources on a per group basis. The only thing that i don't like, is mixing in actual content (the CDATA) into the xml. It's not that i object to the location of the CDATA, it just makes for some ugly looking xml. It's really just a minor nit, and at this point, the positives far outweigh the negatives. Nice work. How do you want to proceed with this improvement? -Original Message- From: Noel J. Bergman [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 28, 2003 8:17 PM To: James Developers List Subject: RE: XMLResources and demo mailet > I haven't gotten the responses.properties table in there They are present in the revision attached. My expectation is that the groups would be used to organize messages for each command, and the "for" matching would be used to allow a particular mailing list to replace one or more message resources. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Shutting down James
> I am working on JMX stuff right now. My objective is to provide > fine-grain redeployment of an appliance inside a container. This > means that all components would be redeployable (without changing > existing component code). Great. :-) > this will not bne available tommorow Understood. We haven't been rushing at JMX. When the support is there from Avalon, we'll want to take advantage of it. If Steve has some stuff already working for James, that'll be good to look at, too. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: PreparedStatement and performance
> Why did Serge use PreparedStatements in eg > JDBCMailRepository.store(MailImpl mc)? You almost always want to use PreparedStatement with arbitrary data. The positional parameters are not parsed as part of the SQL statement, and therefore cannot be misinterpreted as part of the SQL statement. Servers accepting externally provided data and building query strings are typically ripe for exploit. See also: mysql.PreparedStatement.execute() [mysql.PreparedStatement.fillSendPacket() in Connector/J v3] With respect to your performance observations, in a connection pooled environment, a statement pool is necessary for performance, since it is likely that a PreparedStatement will used many times, but not during the scope of a single use of a pooled connection. In Connector/J v2, the statement cache code was incomplete and disabled. In Connector/J v3.0 and v3.1, the statement cache was completely removed. You can ask Mark about why Connector/J does not implement a statement pool. He may be assuming that pooling will be provided external to the driver. See also: JDBC v3 Specification, Section 11.6 http://www.mysql.com/doc/en/C_API_Prepared_statements.html http://archives.apache.org/eyebrowse/[EMAIL PROTECTED] pache.org&msgId=434537 http://jakarta.apache.org/commons/dbcp/apidocs/org/apache/commons/dbcp/Pooli ngConnection.html http://www.oop-reserch.com/poolps_3_0.html FWIW, there are no plans to add a statement pool to mordred, but DBCP has one. --- Noel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Shutting down James
Noel J. Bergman wrote: Well, we need to shutdown the container/application. What is the recommended means? Hopefully there is a means to do this from outside of James. We provide a telnet admin interface. You've never seen it? Ideally we can replace it with a JMX interface. - shutdown: take down the entire system That's the topic at hand. - redeploy: decommission and recommission incorporating any changes to deployment information and/or resoruces As far as I know, Phoenix doesn't support that idea, but we'd like to be able to restart parts of James with a new configuration without having to shut it down. For example, if I've changed the mailet configuration, I'd like to restart the spooler. I should not have to shutdown the SMTP server, since it should not be impacted. Just for reference ... I am working on JMX stuff right now. My objective is to provide fine-grain redeployment of an appliance inside a container. This means that all components would be redeployable (without changing existing component code). Given a hierachy of managers this would enable selective subsystem redeployment. But this will not bne available tommorow - things are currently focussing on sensible and intiative presentation of JMX information. Cheers, Steve. Phoenix has a JMX access point and I belive that MX4J supports SSL based connections (going on memory here) - so in pricipal this should be possible. We disabled MX4J when we found out that it was permitted unsecured administration. I'd like to re-enable. Hopefully Steve will help us get that all square. --- Noel -- Stephen J. McConnell mailto:[EMAIL PROTECTED] http://www.osm.net Sent via James running under Merlin as an NT service. http://avalon.apache.org/sandbox/merlin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tests of mail list.
Hello. I've been testing the James v2.1.3 for supporting a lot of users in mail list. The tests were done for next environment and settings: - all users of mail list are local(their account is contained in this James server) - the Database repository is used. The database is McKoi 0.94h. - the JVM is started with settings -Xms256m -Xmx1024m. In default (64Mb) the Out of Memory occurs for 7500(and more) users in mail list - in settings - 2 The results are: number of users | delivery time | the speed | peak memory usage 500 | less 50 sec | ~10 users/sec | 117 132 Kb 1000 | 1 min 30 sec | 11.1 users/sec | 210 868 Kb 2500 | 4 min 11 sec | 10.0 users/sec | 280 496 Kb 5000 | 10 min 04 sec | 8.3 users/sec | 283 112 Kb 7500 | 24 min 40 sec | 5.1 users/sec | 282 968 Kb 1 | 38 min 04 sec | 4.4 users/sec | 282 580 Kb 15000 | 1 h 16 min 43 sec | 3.3 users/sec | 325 700 Kb 1) Why does the speed of delivery decrease after 2500 users? 2) Why does James use such big memory for sending letters? 3) Also I see that in table spool in recipients field the ALL emails of mail list are placed. May be better for delivery to cut this to a little lists (up to 1000 or 2000 users) and send them separately? I'm interested in correction of this and I can try to fix this. Is it possible? Eugene Best regards, Eugene mailto:[EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: PreparedStatement and performance
I've tested this design with a PreparedStatement and a SingleStatment. With
the design used in James adding a user to a MySQL-DB in
AbstractJdbcUsersRepository takes about 450 ms on my machine comparing 10 ms
with a SingleStatement.
Each time lets say AbstractJdbcUsersRepository.doAddUser(User user) is
invoked the PreparedStatement is (seems to be) recompiled which consumes
much more time than a SingleStatement. In the method the Statement is
executed only one time which is the worst case of using a PreparedStatement
(as my sources told me).
Here the general approaches Ive tested:
First the scenario that should gain performance over a SingleStatement:
create PreparedStatement ps;
while(many times) {
setValues of ps;
execute ps;
}
Second the standard implementation of james:
method {
create PreparedStatement ps;
setValues of ps;
execute ps;
...
}
As you can see the first way creates a PreparedStatement only once and
executes it many times with different Statements.
The second way creates a PreparedStatement each time the method is invoked
and executes the Statement only once.
A strange behavior i encountered was that neigther the
"org.gjt.mm.mysql.Driver"-Driver nor the "com.mysql.jdbc.Driver"-Driver
improved performance compared to a SingleStatement.
- Original Message -
From: "Serge Knystautas" <[EMAIL PROTECTED]>
To: "James Developers List" <[EMAIL PROTECTED]>
Sent: Thursday, May 29, 2003 3:20 PM
Subject: Re: PreparedStatement and performance
> typedef_lex wrote:
> > Why did Serge use PreparedStatements in eg
> > JDBCMailRepository.store(MailImpl mc)?
>
> Because you have to store binary data. Besides, prepared statements
> should be much faster assuming your driver does prepared statement
pooling.
>
> --
> Serge Knystautas
> President
> Lokitech >> software . strategy . design >> http://www.lokitech.com
> p. 301.656.5501
> e. [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
