Re: IMAP server crashes after some time - looking for cause

[Josip Almasi]

On 10/10/2014 04:24 PM, mathias.eck...@t-systems.com wrote:


Hello Team,

we are currently experiencing the same issues as described already here:
http://www.mail-archive.com/server-user@james.apache.org/msg14761.html

We are running james server beta 4 on a linux server with a oracle or a mysql 
database.
After some time the we get timeouts when trying to connect  with imap.
The occurrence seems to be by chance.
We couldn't reproduce the issue yet with tests (which is very important for us 
at the moment).


What kind of tests?


Does anybody know under what circumstances this issue occurs or what concrete 
activity triggers this issue?


But in the mail archive link you provided, Eric Charles answered it's 
solved in beta5:)


Regards...

-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Thunderbird Compatibility with JAMES 3.0?

[Josip Almasi]

On 06/10/2014 10:52 PM, Jerry Malcolm wrote:

Has anyone encountered problems using JAMES 3 IMAP with Mozilla
Thunderbird client?  It appears to work fine with small-medium mailbox
repositories.  But I have a client that has a large mail account with
over 500,000 emails spread across a series of folders, and we are seeing
some serious problems.  On Thunderbird, it sometimes takes 10 minutes to
open the INBOX folder, and then when it does open and you select an
email, it may take another 5-10 minutes to show the email.  This doesn't
happen all the time.  But it happens on several Thunderbird clients. The
inbox usually only has 100-200 emails in it, but there are quite a few
mail rules that run on the INBOX in Thunderbird.  I am configured to
access the same user account via my iPhone.  When Tbird is sitting there
with the circle spinning forever, I can still open the folder and view
the email on my iPhone with no problem.  So I don't think this is a
JAMES problem.  But I'm curious if there are others using JAMES that are
also using Thunderbird and, if so, are you seeing any similar behaviors?


It's mozilla issue, tied to mbox store. You might want to activate 
maildir store, it's supposed to be faster.
Also, check 'compact folders' thing, make sure it's in 'ask' mode, maybe 
it correlates.



Secondly, and more critical, moving a few emails between folders in
Tbird has never been any problem at all.  However, I have tried several
times moving a few hundred emails as a block from one folder to another
folder and also trying to delete a large block of emails (send to
trash).  With larger sets of email, it goes into an infinite loop and by
the time I notice what is happening, I have 30-40+ copies of every email
in the target and it just keeps growing.  I understand that 'move'
really means 'copy' then 'delete' when everything has been copied.  Copy
is looping/restarting and Delete never happens.  One time I  started it
up and unfortunately walked away.  Came back and had like 20,000 emails
in the target folder.  Fun to clean that out, and the only way I could
do that was under-the-hood database work.  Ugly.


Yep that's bad, and I'm afraid it's not mozilla issue.
Anything in james logs?


if JAMES is returning an error on some invalidly formatted email or
something that is causing Tbird to go crazy.  If that's the case,
perhaps I could add defensive code to JAMES to make Tbird not get so
irritated, etc.

I'm going to be pursuing this on the Thunderbird forum as well since it
is likely more Tbird issue than JAMES issue.  But at this point, it's
neither a Tbird nor a JAMES issue... It's MY issue with my client.  So
I'm just attacking all angles.  I'm just looking for information and
more data if others have seen this.


Then it's up to you to analyze logs and protocol.


Anybody want to talk about Thunderbird and JAMES 3?


Yep, please keep us posted.

Regards...

-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Using Apache James for Reputation Management in Email Systems

[Josip Almasi]

Hi,

for a while now I have james rep system on my wish list.
Did some research, and was going to implement it... soon... just after I do 
this and that... and so it goes on for years:)
So here are my ideas about, maybe you can use them somehow.

NeuroGrid [1] is a neat yet simple reputation system, works with simple 
positive and negative stimulus.
Negative stimulation is received when user gets recommendiation, and positive 
when user confirms it.
In web environment, that would correspond to receiving search results, and 
clicking on a link.
In mail environment, corresponding negative and positive stimulus is 
downloading envelope, and downloading message body.
Note this applies to IMAP only.
Furthermore, additional positive stimuli apply when a reply is sent; SMTP only

In JAMES environment, you'd need to
- extend IMAP server
- extend message store
- write you own mailet
- possibly, extend user store

And you don't need to customize email client.
Of course, it may be nice to query users reputation, but that need not be 
integrated into email client - a web app will do.
As for standard operation during reading of mails, usual subject changes would 
do, i.e.
[troll 42.03%] original subject here
[guru 99.98%] and the subject
etc.
Once you get that working, it's easy to do the rest with standard mail filters.

That's in short about my would-be rep system, see if you can use any.
Also, I would love to learn more details of yours.

Regards...


[1] http://www.neurogrid.net/
Aww the site is down:(
Look it up on SourceForge, and check p2p reputation papers by Sam Joseph et al.


Dileepa Jayakody wrote:

Hi All,
I'm Dileepa a newbie to Apache James. I'm doing my Msc research project on : 
Reputation Management in Email Networks in which I aim to implement a 
reputation management network among peers using email systems.
Each user will have an index of reputation scores for his contacts (based on 
email content analysis as per his personal context and number of 
spam/not-important messages sent by the contact). This reputation scores should 
be shared among peers in a
controlled manner (ACL, authorization) to deliver a collaborative reputation 
network. I'm planning to use an extended SMTP protocol to share reputation 
attached to email users.

I came across the Apache James  project and realized it 
provides an API to extend existing protocols like SMTP and IMAP to perform additional tasks 
via APIs like James Mailet  and SMTP
Hooks.

Below is the architecture I have in mind for reputation management via email 
networks. Reputation Server, ReputationBox are analogous to the email IMAP/POP 
servers and MBox of users. I highly appreciate your ideas on my project and 
wish to incorporate your
suggestions on using Apache James or related technologies to my project.

Your thoughts and tips on James are highly appreciated.

Inline image 1
Regards,
Dileepa



-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Bayesian Analysis for v3

[Josip Almasi]

David Legg wrote:

That's pretty straightforward actually.  Suppose you have a sentence "Mary had a 
little lamb" then you would generate the following token values in addition to the 
single word tokens if you were capturing a phrase size of 2: -

   Maryhad
   hada
   alittle
   littlelamb


Neat trick, I wonder how it works out.
Might be too large, especially with malformed MIME types.


I recommend you read Paul Graham's 'Better Bayesian Filtering' [2] (especially 
the bit titled 'Tokens').  It's fascinating stuff... or maybe I'm getting too 
old and geeky :-)


Sure I did, quite a while ago.


Image info needs extracting too.  So things like the width, height, bit depth, 
type of encoding, Exif data and any tags should all be captured.


...what would you use to extract image info?


I haven't used any graphics libraries recently but a quick scan suggests 
'Commons Sanselan' [3] which happily is an Apache project now.


Seams easy.
Broken link to MedatdataExample.java:/

Well, you got it all covered.

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Bayesian Analysis for v3

[Josip Almasi]

David Legg wrote:

Hi Josip,

Thanks for your comments.

On 24/10/12 15:42, Josip Almasi wrote:


I think I'll wait till it works with java 7. (workaround didn't work for me)


I didn't know that.  I'm Ok with Java 6 for the moment as that is the default 
with Ubuntu 12.04.  Still not quite comfortable with this iced tea business 
though... I prefer 100% Java beans :-)


Well, new JAXB broke more applications. Right now I can't remember exactly 
which ones, but I had to go back to JDK 6.


So my first plan is to make the tokenizer more intelligent.  It should 
carefully extract far more meta-data from the email.


Wrote some mail parsing code, parses plain text and html, ignores other MIME 
types. For others, I guess only headers should be taken into account.
Malformed MIMEs are real issue there. So I used heuristics to avoid them - 
number of tokens and size of tokens.
Also, better ignore numbers, or use them as delimiters.
Of course, all message parts need to be processed. That's not cheap, and should 
be limited, by max allowed time and/or number of tokens.


That's very interesting.  Did you use the Mime4J library to do the heavy 
lifting or did you parse all the message yourself?


I used javax.mail, started from a good mail parsing example included.
Parsed html with javax.swing.text.html.HTMLEditorKit.

It's for my mail archiver, not (yet) having anything to do with JAMES:
http://sf.net/projects/mar

So I did sort of opposite of what antispam is intended to do: I captured only 
'good' keywords.


That's a good point about malformed MIMEs.  Even with the relatively small 
number of spams I've collected I noticed a number of deviant practices.


Tell me about it, one even managed to produce StackOverflowError in html 
parser:>


Not so sure about ignoring numbers though.  Certainly, need to capture IP 
addresses, HTML and CSS colour settings and also domain names.  I can see there 
will be a lot of tweaking involved.


Ah CSS, I forgot about it completelly. True, has to be analyzed.
Uh, HTML... right, for antispam purposes, tags need to be saved too.

The catch with numbers is, I recieved some CSV files, containing database table 
dumps, hundereds of thousands of lines, each containing unique codes.
And of course, many many smaller ones, with various server logs etc.
Best being left alone.

IP and domain names, I don't think so.
Suppose you use dot as delimiter. Then, each byte of IP address becomes a 
token, and gets own weight. Much the same with domains.
Bayes should take care of rest.
IP addresses are relatively rare in mails, domains being much more important.
Now, should we tokenize www.spammer.com, then weight www, spammer, and com, or 
should we store domain as it is?
I think - tokenize.
It's just a bit more processing, but possibly much less storage:
- one "www" and "com" instead of zillion stored
- two dots less
- "spammer" is just another keyword stored, weighted, possible to occur in 
other mails containing no domain www.spammer.com
(this is all about message content of course, headers should not be tokenized)

Anyway, here's my delimiter list:
" ,./<>?`~!@#$%^&*()_+=-{}|[]\\;':\"\r\n\t1234567890"
Though numbers should probably be excluded:)
Watching parsing time and keyword number should eliminate problems with numbers.


I'm keen to capture phrases (ie. capturing two or more sequential words) as 
I've heard they improve detection at the expense of a larger token database.


Any pointers?

I don't know... quite complicated.

Though some lexical comparison might make sense. Here I wrote some examples, 
but that got 7.1 spam score and was returned back to me:)


Image info needs extracting too.  So things like the width, height, bit depth, 
type of encoding, Exif data and any tags should all be captured.
I quite often get large (several megabyte) emails from China containing 
pictures of products for me and the
current James setup gives up with messages of that size.  Or rather it creates 
thousands of random tokens full of base64 segments!


That's interesting, I don't get these. At least not as single part messages. So 
bayes probably picked up other keywords of text/html part, and headers.
So I think that's too much effort for small gain.

Anyway, what would you use to extract image info?

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Bayesian Analysis for v3

[Josip Almasi]

Hi,

David Legg wrote:

Hi all,

It's been a long time since I frequented this list!

After many years of faithful service I'm upgrading my server and thought I'd 
check to see what's happening with James.  I'm pleased to see v3 is beginning 
to emerge and I'll be happy to take it for a spin.


Same here. Though I think I'll wait till it works with java 7. (workaround 
didn't work for me)


I see nothing much has changed with the Bayesian analysis mailet. It has 
performed very well for me and I'd definitely recommend it to people. However, 
I've just taken a look at the code for the first time and I think I'd like to 
have a go at improving it,
especially as IMap is now a possibility.

I have a couple of ideas I'd like to try and I thought I'd air them here in 
case anyone has a brighter idea or some advice; thanks.

As it stands, the current Bayesian filter has a relatively simplistic 
tokenizer.  It literally seems to break the email into tokens with little 
regard to whether that bit of text is a mime boundary, base64, image, document 
or header etc.  My spam and ham
database is filled with millions of random looking chunks of text mainly from 
base64 encoded images!  So my first plan is to make the tokenizer more 
intelligent.  It should carefully extract far more meta-data from the email.


I might help you with that.
Wrote some mail parsing code, parses plain text and html, ignores other MIME 
types. For others, I guess only headers should be taken into account.
Malformed MIMEs are real issue there. So I used heuristics to avoid them - 
number of tokens and size of tokens.
Also, better ignore numbers, or use them as delimiters.
Of course, all message parts need to be processed. That's not cheap, and should 
be limited, by max allowed time and/or number of tokens.


I'm not the first to think of this of course.  Paul Graham originally wrote 'A 
Plan for Spam' [1] back in 2002 and then updated it with 'Better Bayesian 
Filtering' [2] in 2003.  This spawned several projects and products.  The more 
feature complete version
is SpamProbe [3] by Brian Burton but a Java version exists with a project 
called jASEN [4]. This latter project has been quiet for a few years and was 
forked into a proprietary product as well.

I'm quite interested in the fact that James 3 supports IMap.  I think this may 
make it easier and more efficient for user's to maintain their own spam folder. 
 Currently user's have to send any spam (or ham) they receive to an address 
such as s...@xxx.yyy
(or no-s...@xxx.yyy) and if they forget to send it as an attachment they risk 
poisoning the spam corpus.  Think how much easier it would be to simply move an 
email from one of your email folders to a special 'spam' folder.  Also think 
how much easier it
would be to browse the spam folder looking for mis-classified emails and drag 
them back to the correct folder. Currently, I delete emails classified as spam 
and if someone wants it back I have to go rooting about in MySQL's binary logs!


Right!


I worry how big the spam folder may get if I'm not deleting spam messages.  I 
may have to automatically expire spam messages that get to a certain age.  Or 
it may be that a small amount of fastfailing reduces the spam intake to 
manageable amounts.


Well, I'm not deleting any spam:) You never know when you may need some;)
Right now I have 143286 unread in my junk folder, total is 250k+, all correctly 
marked as 100% spam, 850MB.

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

ANN: a stress-test tool

[Josip Almasi]

Hi all,

I wrote a simple stress-test tool:
http://sf.net/projects/spizd

Supports smtp, pop3 and imap, plain, tls and ssl.
And more, but others are off-topic for this list;)

It's primary purpose is to test how many connections servers can handle.
It can also send mails, thus might be used to measure throughput.

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Bottleneck on James spool storing code

[Josip Almasi]

rafael.munoz wrote:

Hello


Josip Almasi wrote:

rafael.munoz wrote:

Hello

I have been doing some stress and performance tests on James 2.3.1 and I
think I have found a bottleneck on the James spool storing code (I am
using 
the filesystem spool). 


I have configured James to behave as a simple SMTPServer and do almost
nothing more than receiving mails and storing it in the spool ("match="All" class="Null"/>"). 

Eh, Null?



I was only measuring James input so I was just destroying any incoming
message after retrieving from the spool. 'Null' is refering to the
NullMailet (http://james.apache.org/mailet/standard/mailet-report.html#Null)


Well I know but then maybe those complexity things I said don't apply 
here; they generally apply to directories with large number of files.





Josip Almasi wrote:

...

So, summarizing:

...

2. Anyone knows why the FileOutputStream object creation takes more and
more
when James is stress out? The underlying OS is not reporting any problems
with the filesystem or the file descriptors.
Most filesystems store directory entries as lists. Lists are read each 
time from beginning so to access Nth entry you'll access N-1 entries, 
IOW N*(N-1)/2 complexity.
In fact, AFAIK only FS that doesn't do that is ReiserFS, and I doubt you 
can get that on solaris, so better switch to database storage.
Databases use ballanced trees meaning IIRC max N*log(N) avg log(N) 
complexity.


Umm .. interesting, I didn't know that. I would check the number on entries
on the spool directory when I start to get huge file FileOutputStream
creation times (that as you implies almost surely are linked to huge file
creation times on the filesystem). And about the database suggestion, I am
afraid it is not an option in our application :(.

Thanks for your answer!


Welcome.
Furthermore, if you use FS, JAMES has to keep list (or map) of messages 
in memory. For lists and treemaps, above complexity applies.
Plus, memory usage... spammers kindly provided me with even better 
stress test in production:> Check this thread:

http://marc.info/?l=james-user&m=121491652506688&w=2

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Bottleneck on James spool storing code

[Josip Almasi]

rafael.munoz wrote:

Hello

I have been doing some stress and performance tests on James 2.3.1 and I
think I have found a bottleneck on the James spool storing code (I am using 
the filesystem spool). 


I have configured James to behave as a simple SMTPServer and do almost
nothing more than receiving mails and storing it in the spool ("match="All" class="Null"/>"). 


Eh, Null?

...

So, summarizing:

...

2. Anyone knows why the FileOutputStream object creation takes more and more
when James is stress out? The underlying OS is not reporting any problems
with the filesystem or the file descriptors.


Most filesystems store directory entries as lists. Lists are read each 
time from beginning so to access Nth entry you'll access N-1 entries, 
IOW N*(N-1)/2 complexity.
In fact, AFAIK only FS that doesn't do that is ReiserFS, and I doubt you 
can get that on solaris, so better switch to database storage.
Databases use ballanced trees meaning IIRC max N*log(N) avg log(N) 
complexity.


Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: myhost.com?

[Josip Almasi]

Dockery, Michael E SFC NG NG NGB wrote:

I keep finding these messages in my logs
 regarding "myhost.com" (see below)


*
17/12/08 05:15:34 INFO  James.Mailet: RemoteDelivery: Attempting 
delivery of Mail1229054558354-2-to-myhost.com to host myhost.com at 
72.89.160.195 for addresses [my_addr...@myhost.com]


17/12/08 05:15:56 INFO  James.Mailet: RemoteDelivery: Could not connect 
to SMTP host: 72.89.160.195, port: 25



Well either myhost.com is resolved but not listening, or james can't 
connect to your smtp gateway.


j...@jozo:~> ping myhost.com
PING myhost.com (72.89.160.195): 56 data bytes
64 bytes from 72.89.160.195: icmp_seq=0 ttl=41 time=125 ms
64 bytes from 72.89.160.195: icmp_seq=1 ttl=41 time=109 ms

:)))

Well it's resolved allright:)

j...@jozo:~> telnet myhost.com smtp
Trying 72.89.160.195...
telnet: Unable to connect to remote host: Connection timed out

... and it's not listening.


any ideas
 where these message are coming from
  or why they are being generated?


Well someone sent a mail to my_addr...@myhost.com.

Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: Relayings

[Josip Almasi]

Sandeep Giri wrote:

Hi,I am trying to setup James such that it recieves all the mails to
james.mydomain.com and should be able to send outgoing mails through server
X.

James is able to successfully recieve mail but is not sending outgoing mail
because it is trying to lookup mydomain.com first and if it is not able to
find the MX records of mydomain.com it is failing to deliver rather than
trying to deliever throught the server X which I have mentioned 
tag.

Any ideas?


Well this seems like correct behaviour to me.

Why would you want to send from fake domain anyway?

Maybe you can try checkValidSenderDomain in james-smtphandlerchain.xml 
but I doubt that's it.


Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Re: 'better bayesian' issue

[Josip Almasi]

Jerry M wrote:
Or you can use SPF if you set up your DNS server with an SPF record for 
your domain.  If the email is from "you", then SPF will catch it.  I use 
SPF along with the Bayesian filter.  SPF catches it every time if it's 
"from" one of my domains.


Well that's a neat trick, thanks Jerry.
I suppose I could add a SPF record.
Any more pointers on now to implement it in James?
Right now I'm looking at james-smpthandlerchain.xml but I don't 
understand how to enable SPF check... checkAuthNetworks?


Regards...


-
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

'better bayesian' issue

[Josip Almasi]

Hi all,

seems spammers read Paul Graham too:)
I'm getting some spam that simply won't get any spam rating.
It's a html, displaying a single image, alternate a single line pointing 
to a single url, usually saying "Click here to view as a webpage".

However it's always different pic and different site.
As for message attributes, nothing usefull there either.
From and To is me and my legitimate users:) Mostly this email, it's 
been around 10 years I bet every spamer has it...


So, how to fight it?

TNX

Regards...


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: spammers DoS my James

[Josip Almasi]

Josip Almasi wrote:


Trying to think of problems - it may make problems to MTA that opens a 
new connection for each msg. But of course there's no such MTA's 
around;) But in such a case this method is equivalent to greylisting.


Well as I suspected but didn't dare say, qmail is sensitive to this method.
It's still OK for regular mail traffic, but mailing lists delivered via 
qmail result in bounce test msgs.

And apache.org delivers with qmail, including this list:>
Well it's not a big deal really, we need only manually whitelist 
apache.org and other Bernstein fans:)


Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Mail marketing with James Mailet - sending Millions mails

[Josip Almasi]

Bruno Pedrosa wrote:

Hi David, It isn't for spamming the world. lol
We have some clientes like stores and restaurants that want to send e-mails
with news and promotion for their subscribed clients. All e-mails are
legitms and can be unsubscribed at any time. 


Well now that you say that...;)
Seems you need usual mailing lists.
James handles that, though I didn't use it yet.
So have a look at james-listmanager.xml and try playing with testlist.

But your performance requirement seems unrealistic. Seems like someone 
took that requirement from sms/mms delivery sofware:) Mail delivery 
includes name resolution, you know, that DNS MX thing... say, I just 
resolved your address and it took 1.043 secs first time and 0.016 secs 
after it was cached.


HTH

Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: spammers DoS my James

[Josip Almasi]

David Legg wrote:


Mucking about with iptables always brings me out in a cold sweat... one 
slip and you can be locked out of a remote server :-)


Hehe I know the feeling;)

I'm not sure if it would be effective though if a spambot stays 
connected while doing a dictionary attack or while it tries to relay 
several hundred emails once it has cracked an account.  For that, James 
itself would have to keep count of the number of commands executed and 
force a disconnect after some threshold is reached.


Usually a few seconds timeout is just fine protection against bruteforce 
attacks. It may not be as good against dictionary attacks, but its up to 
admin to disallow weak passwords.

So IMHO auth handler should just sleep a a bit after unsucessfull auth.
Then again, that's why attacker does not wait for the response, he just 
opens another socket and tries again:) And thats where the iptables 
trick kick in;)


Once cracked, well, theres not much protection agains that.
FTR I see there's some sleep interval in RCPT handler...

You seem to be suggesting that the same IP address is connecting a lot 
of times.  


Right. Seems like someone modified ssh attack to smtp:)

If I were a spambot writer I would be trying to get as many 
messages relayed as possible and re-connecting each time I tested a 
password or sent a message would get in the way of that.


Sure. Just dont tell them that;)

Regards...


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: spammers DoS my James

[Josip Almasi]

Josip Almasi wrote:
I wonder if something similar could be implemented here?  [1] 
http://olivier.sessink.nl/publications/blacklisting/index.html


Gonna give it a try right away;)


Well it works allright:)

Here's the script modified for smtp traffic(*).

Trying to think of problems - it may make problems to MTA that opens a 
new connection for each msg. But of course there's no such MTA's 
around;) But in such a case this method is equivalent to greylisting.


Thanks again for the hint:)

Regards...

*)

#SMTP dynamic blacklist impl
#based on SSH blacklist
#http://olivier.sessink.nl/publications/blacklisting/index.html

# create properREJECT chain that does different rejects for tcp/udp
iptables -N properREJECT
iptables -A properREJECT -p tcp -j REJECT --reject-with tcp-reset
iptables -A properREJECT -j REJECT --reject-with icmp-port-unreachable
#
iptables -N blacklistdrop
iptables -A blacklistdrop -j LOG --log-prefix "adding to BLACKLIST: "
iptables -A blacklistdrop -m recent --name BLACKLIST --set -j DROP
#
#
# on external hosts, do rate limiting on incoming smtp packets, and keep 
a black list for 30 seconds

# this rule drops *any* packet if the IP is in the blacklist
# icmp 'destination-unreachable' packets should not update BLACKLIST, 
because

# they are generated by our own REJECT rule in the extern_out chain
iptables -A INPUT -m recent --name BLACKLIST --update --seconds 120 -j DROP
#
# all *established* smtp connections simply continue
iptables -A INPUT -p tcp --dport 25 -m state --state ESTABLISHED,RELATED 
-j ACCEPT

#
# *new* smtp connections are all put into a list 'smtpconn', and if 
there are 3 such packets in 30 seconds
# we send the package to chain 'blacklistdrop' which puts the IP in the 
blacklist
iptables -A INPUT -p tcp --dport 25 -m state --state NEW -m recent 
--name smtpconn --rcheck --seconds 30 --hitcount 3 -j blacklistdrop

#
# if we have seen less then 3 such packets in the last 30 seconds we accept
iptables -A INPUT -p tcp --dport 25 -m state --state NEW -m recent 
--name smtpconn --set -j ACCEPT

#
# if the destination address is in the blacklist, we REJECT *any* packet
iptables -A OUTPUT -m recent --name BLACKLIST --rdest --rcheck --seconds 
30 -j properREJECT

#
# outgoing we accept all smtp traffic, with connection tracking
iptables -A OUTPUT -p tcp --sport 25 -m state --state 
ESTABLISHED,NEW,RELATED -j ACCEPT


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: spammers DoS my James

[Josip Almasi]

David Legg wrote:


Oh, bad Luck :-(


Indeed:)

Long ago I decided to 'trust the force' and delete anything which failed 
the bayesian filter.  That stops the backlog of messages for me but that 
may not be acceptable for you.


Well I solved the issue by droping all address-error msgs.
But I'm still amazed by incredible ammount of msgs they can produce...:)

A while ago, like many people, I noticed those horrible scripts 
attempting dictionary attacks on the SSH daemon.  In the end I 
implemented a fix I'd seen using iptables [1][2].  This involved 
dropping any attempts to login using SSH if that same IP address had 
previously failed to login for more than some threshold value.  The ban 
on the IP address was set to half an hour.


I wonder if something similar could be implemented here?  
[1] http://olivier.sessink.nl/publications/blacklisting/index.html


Great, thanks!!!
Gonna give it a try right away;)

Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Correct Heap Space Settting for JAMES?

[Josip Almasi]

Jerry M wrote:
I'm getting a ton of Out of Memory errors in James in the connections 
log. (James 2.3.1)  I've been running James for about 4 years and am 
just now beginning to hit these Out Of Memory errors.  I don't see any 
significant changes to the traffic patterns for mail, other than the 
ever-growing spam count.  But spam shouldn't be causing OOMs, should 
it?  


Well it should not, but seems it does latelly (check spammers DoS my 
James thread); I suggest you remove these spam msgs.


But regardless of whether I can identify why I'm getting the OOMs, I 
have to fix this.  Where do you set the heap space allocation for 
James?  


In startup script.
But it won't help long term you know...

I'm assuming the 'recommended' value is whatever the default is 
set to.  But obviously that's not working for me.  So what is the 
maximum safe value I can set the heap size to?


With 32-bit jre its somewhere between 1.6-1.8GB, with 64-bit much more.

Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

spammers DoS my James

[Josip Almasi]

Hi all,

recently, I've noticed my server runs a bit slow.
Netstat shows me I have a dozen smtp connections from a dozen adresses, 
from bulgaria and russia and china, I thought they try to brute force 
crack my smtp auth so I just iptables them away.
But turns out they change addresses and they brute force guess my 
server's usernames:)
So my james died with OutOfMemoryError, after 8GB spams in address-error 
directory, about 1.8 million messages, when 500 MB ram wasn't enough.
FTR I did receive 38468 spams (~2%), which bayesian server & client side 
correctly identified as spam and stored to my junk & trash folders; I 
use them for statistics, want more statistics let me know.


Makes me think, james default conf is in fact insecure - spammers may 
DoS your server away anytime. They just bomb you with millions of 
messages, you never read any of them, but your server dies.


Should't default matcher class be Null for address-error?

Does database store help at all? You just get db or disk full instead of 
memory full, right?


BTW, AFAIK ppl use greylisting to rather sucesfully get rid of spammer's 
DoS. Here's how it works: spammer gets 450 service temporary unavailable 
and gives up, and good MTA retries after a while and delivers.
Personally, I'd never use it, it only introduces unnecessary delays in 
mail delivery; exact delay depends on foreign MTA config, and it can be 
quite annoying in biz environment. Furthermore, it's just a matter of 
time when spammers get smarter and greylisting won't work anymore.


Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: How to embed james in an Java server application

[Josip Almasi]

michael wu wrote:


Hi,
I am thinking to embed james mail server in a server applications of
mine.  Had anyone done this before?  Can anyone point me to any
reference?


Yep I did, in fact I run loads of java cra^H^H^Hsoftware in one virtual 
machine. I use reflection to start james so I don't have to link with 
it. Here's the code:

http://vrspace.cvs.sourceforge.net/vrspace/vrspace/src/main/org/vrspace/server/object/James.java?revision=1.3&view=markup

I didn't try it with latest release yet, please let me know if you do.

Regards...

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: RE : i need help urgently

[Josip Almasi]

Nathan Cheng wrote:
So a message like this in config.xml would haved saved me a lot of trouble:
"Use of mordred with Sun Java 1.4.2_03 may result in a deadlock under 
some circumstances; use of DBCP in these same circumstances will make 
your deadlock go away."
Easy to say _now_, but if we knew that before, you'd be able to learn it 
from this list. Comments like this would soon grow into huge useless 
file noone would ever read. I.e.
- as of j2sdk 1.4.2_05, Sun has finally fixed ClassLoader.loadClass() 
method by adding checkName() method. Since ClassLoader now works 
according to the specification, code written for j2sdk earlier than 
1.4.2_05 may fail with strange ClassDefNotFoundError messages. If you 
experience this behaviour using Sun java, please downgrade to jdk 
1.4.2_03 or earlier.
- jdk v this on hpux v that was linked with zlib that may produce buffer 
underrun. Although no such exploits is known yet, please upgrade to jdk 
v something before someone breaks into your network by mailing you a 
picture.

Etc... IMHO this list is _much_ better, and provides me (and as we've 
just seen, you too) with much better tech support level than any company.

Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: NNTP Server plans and usage

[Josip Almasi]

Chemi wrote:
Thanks for the info Josip.
BTW, I have visited your web (http://vrspace.org/) and I have found: 
http://vrspace.org/forums/index.php which is a web forum. Does this mean 
you have two different solutions for forums? Or you have synchronized 
both access in a single repository: HTTP and NNTP?
In fact we have _three_ solutions:)
Developers collaborate over mailing list provided by sourceforge. SF 
mailmain feeds our news server (JAMES), ml archive published over news 
is better than sf ml archive in many aspects, i.e. anonymous access, 
archived binary attachments etc. Forum software is written in php and I 
still see no way to integrate. Forums are nice for support and general 
discussions, i.e. users often get confused with mailing list 
subscriptions etc.
I _would_ like to have lists, news and forums integrated, but having 
them separated doesn't hurt much, as each of them have advantages for 
some purposes. BTW joelonsoftware.com had an excellent article on forums 
vs news vs lists, but it probably isn't accessible online.

Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: NNTP Server plans and usage

[Josip Almasi]

Chemi wrote:
Hi, I would like to know any info about this points:
- Do you use James as NNTP Server in production anywhere?
Yes: news://vrspace.org/org.vrspace.dev
- Which are the plans for NNTP Server wihtin James? James website states 
that NNTP is experimental. Is known when it will be stable.
My experience says it _is_ stable.
There's still one patch recently posted here, so my guess is next build 
will work just fine.
Can't comment on your 'delay' issues, it's not an issue for me.

Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Problem reading posts from NNTP server

[Josip Almasi]

Steve Brewin wrote:
IMHO wrong: what's the point if readers break the rules? Makes server
useless.
Standards are the things which allow complex interactions to take place in
this big wide world of ours. I'm sure you would be upset most times if a
standard was not adhered to - think world wide web, human rights or your
neighbour choosing to drive on the opposite side of the road to the rest of
you. Standards matter.
Yeah I _am_ upset most of the time cuz 90% people use exploiter which 
does not implement standards and web sites don't use standards cuz 90% 
people don't use standards.
And in this case, I'm upset cuz my very standard news server doesn't 
deliver messages sent by non-standard readers.

I have no detailed knowledge of the specific issue, but if someone is
breaking the rules, isn't it they that are 'wrong'?
Well, I'd say so. But again, the point is I need these articles _delivered_.
OK now after I've read the rest of the thread - thanks for patch!
Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Problem reading posts from NNTP server

[Josip Almasi]

Chemi wrote:
More info about this problem. Reading log file: 
...
It seems to me that James doesn't send correctly the end information to 
the client to end the communication. Or Mozilla and Jakarta Common-Net 
doesn't know when the communictaion has ended, but it is 2 against 
1. :-)

Have you seen this behavior before?
Yep. Reader waits till timeout, server never delivers.
Workaround: edit the article and press enter at the last line...
Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Problem reading posts from NNTP server

[Josip Almasi]

Noel J. Bergman wrote:
Guys,
If some of you would like to help contribute to the NNTP handling, that
Well I was asking this on dev list but got no response (news issues).
would be great.  Please note that we stick to the RFC specifications.  In
this case, RFCs 977 and 850 apply.
IMHO wrong: what's the point if readers break the rules? Makes server 
useless.

Regards...
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: worms

[Josip Almasi]

bill parducci wrote:
personally, i like this as a filter:

^[ ]*name\=\".*\.(pif|bat|scr|exe|vbs)\"
Thanks guys,
well I can add only this tip that admin fella gave me:
$bad_exts = '(ade|adp|app|asd|asf|asx|bas|bat|chm|cmd|com|cpl|crt|
dll|exe|fxp|hlp|hta|hto|inf|ini|ins|isp|jse?|lib|lnk|mdb|mde|msc|
msi|msp|mst|ocx|pcd|pif|prg|reg|scr|sct|shb|shs|sys|vb|vbe|vbs|vcs|
vxd|wmd|wms|wmz|wsc|wsf|wsh|\{)';
(yes there's security holes even in hlp)

Regards,
Josip


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

worms

[Josip Almasi]

Hi all,

since Swen.A started spreading on 18th, it removed security software 
from millions of windoze boxes, which resulted in more worms on the loose.
Well I suppose you noticed this:)

How do you defend?

Here's what I did:
- got free antivirus http://clamav.elektrapro.com/
- got antivirus mailet from http://www.mailet.org/
- installed mailet 
(http://nagoya.apache.org/wiki/apachewiki.cgi?James/CustomMailetPackages)

and that mostly did it.
(BTW I got a lot of mails from antivirus software: Hi, I did't send you 
mail beacuse it contains virus. Buy me so you wont get viruses, BUYME 
BUYME!)

Any hints?
Do you forbid your users sending exe pif etc. attachments?
How about worms with wav and other multipart things not in attachments?
Can I use bayesian spam mailet to stop them?
...?
Regards,
Josip


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]