[Assp-test] New version

[Steve Moffat]

Hi
I updated to the new release today and rebuildspamdb has ruined my corpus 
confidence. Not too happy with that

Sep-11-12 16:26:55 Spam Weight:3,904,196
Sep-11-12 16:26:55 Not-Spam Weight:   1,950,092

Sep-11-12 16:26:55 Corpus norm: 2.0021 - (warning: extremely spam 
heavy)
Sep-11-12 16:26:55 Corpus confidence:  0.06224349
Sep-11-12 16:26:55 Recommendation: RebuildSpamDB will limit the number of used 
messages in your corpus. Excess files will be ingored.
Sep-11-12 16:26:55 Corpus norm should be between 0.6 and 1.4

Thanks
Steve
Steve Moffat
Operations Director
Optimum IT Solutions
Desk:   441 292 8849
Mobile: 441 292 8849
MSN IM: st...@optimum.bm
Web: http://www.optimum.bm

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] no sender email in log => ASSP 1.9.7.5(0.0.02)

[Fritz Borgstedt]

ASSP development mailing list  schrei
bt:
>
>From: "Email Marketing" 



ASSP shows the envelope sender in the logfile not the "From:".

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: Re: Antwort: strange ASSP behavior

[Thomas Eckardt]

>ASSP will extract the headers and body and perform
some checks to see if it already "saw" that file

Exactly this way it works for years now - I think we had this topic some 
months ago Andrea - how ever, good ideas comes back in mind every time! 
:):):). 
It must be - because if a mail is reported as ham, it is possibly already 
in the spam folder, but if reported we have to ignore it in the spam 
folder (and via vers spam->ham).
An MD5 hash is calculated over every mail body!

>slightly different content
HMM eliminates this problem!

>for example it may just process
>(consider) the headers
headers are simply too different in terms of bayes and HMM to get good 
results. There is no human language used except the subject. The rebuild 
retrieves some tags from there to get information for the user and/or 
domain based spamdb and hmmdb. How ever, if the body was already seen, 
also the header is ignored.

Thomas



Von:Grayhat 
An: assp-test@lists.sourceforge.net, 
Datum:  11.09.2012 18:02
Betreff:Re: [Assp-test] Antwort:  strange ASSP behavior



 
> I'll explain a bit more:
> 
> - all folders are processed : "the youngest files first"
> - both error folders are fully processed up to MaxFiles
> 
> As the result of processing the first two folders we get a weight 
> (spam/ham). Now we know were we are: we have a current weight, a
> wanted weight, and we now how many files are in the spam and notspam
> folders. Now assp calculates the maximum of files in the spam folder
> that could be apx. used , if we assume that at least all files in the
> notspam folder will be enougth to get the wanted target norm.
> The spam folder is processed.
> Now we know the new spam/ham weigth and can more exactly calculate,
> how many of the files in the notspam folder are required to reach the
> wanted target norm.
> 
> I'm expressed, how exact it was working in my case.

mumble (thinking loud); our problem (if we want to call it so) is that
we may have multiple spam/ham files with the same contents but
different headers or even with slightly different content... now, let's
leave the latter alone for the moment; let's try thinking about those
"similar" files (same body, different headers); in such a case we may
consider some mechanism so that, whenever (storing ? rebuilding ?)
processing them, ASSP will extract the headers and body and perform
some checks to see if it already "saw" that file (e.g. using a DB table
containing hashes or the like) and, if so, ASSP may just avoid
processing the whole "additional file"; for example it may just process
(consider) the headers and skip the body (since it already saw it); I'm
not sure it makes sense, again, I'm just thinking loud here...

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: strange ASSP behavior

[Grayhat]

 
> I'll explain a bit more:
> 
> - all folders are processed : "the youngest files first"
> - both error folders are fully processed up to MaxFiles
> 
> As the result of processing the first two folders we get a weight 
> (spam/ham). Now we know were we are: we have a current weight, a
> wanted weight, and we now how many files are in the spam and notspam
> folders. Now assp calculates the maximum of files in the spam folder
> that could be apx. used , if we assume that at least all files in the
> notspam folder will be enougth to get the wanted target norm.
> The spam folder is processed.
> Now we know the new spam/ham weigth and can more exactly calculate,
> how many of the files in the notspam folder are required to reach the
> wanted target norm.
> 
> I'm expressed, how exact it was working in my case.

mumble (thinking loud); our problem (if we want to call it so) is that
we may have multiple spam/ham files with the same contents but
different headers or even with slightly different content... now, let's
leave the latter alone for the moment; let's try thinking about those
"similar" files (same body, different headers); in such a case we may
consider some mechanism so that, whenever (storing ? rebuilding ?)
processing them, ASSP will extract the headers and body and perform
some checks to see if it already "saw" that file (e.g. using a DB table
containing hashes or the like) and, if so, ASSP may just avoid
processing the whole "additional file"; for example it may just process
(consider) the headers and skip the body (since it already saw it); I'm
not sure it makes sense, again, I'm just thinking loud here...

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: strange ASSP behavior

[Thomas Eckardt]

I'm sorry Andrea - I used the terms a bit confusing.

corpusnorm is not the 100% correct word, if we look at all files in the 
corpus - we should better say, the resulting 'norm' of the spamdb and HMM

In the previouse versions the limit was MaxFiles - so if you had more 
files in the folders, the 'corpusnorm' was not the norm all files - only 
from MaxFiles.
Now assp uses as many as possible (max MaxFiles) , but less as required to 
get the wanted norm ('corpusnorm') for the spamdb and HMM.

Target is, that you don't has to care about the folders  - assp will use 
and/or ignore and/or delete the right files at the right time.

>Hmm... I see now, basically with the latest change you added some logic
>so that older files (which should otherwise be discarded) are ignored
>by the rebuild process... am I right ?

I'll explain a bit more:

- all folders are processed : "the youngest files first"
- both error folders are fully processed up to MaxFiles

As the result of processing the first two folders we get a weight 
(spam/ham). Now we know were we are: we have a current weight, a wanted 
weight, and we now how many files are in the spam and notspam folders. Now 
assp calculates the maximum of files in the spam folder that could be apx. 
used , if we assume that at least all files in the notspam folder will be 
enougth to get the wanted target norm.
The spam folder is processed.
Now we know the new spam/ham weigth and can more exactly calculate, how 
many of the files in the notspam folder are required to reach the wanted 
target norm.

I'm expressed, how exact it was working in my case.

Let's see how it works.

Thomas





Von:Thomas Eckardt 
An: ASSP development mailing list , 
Datum:  11.09.2012 16:55
Betreff:[Assp-test] Antwort: Re:  Antwort:  strange ASSP behavior



>I see, so, basically, you're saying that the weight reported in the
"rebuild report" isn't correct ?

No - the values were correctly shown. But ASSP has used all files (up to 
MaxFiles) even it was better to use some less ( from here or there) to get 

a better corpusnorm.

Thomas



Von:Grayhat 
An: assp-test@lists.sourceforge.net, 
Datum:  11.09.2012 15:25
Betreff:Re: [Assp-test] Antwort:  strange ASSP behavior




> Andrea,

Hi there, Thomas, we are on the public list, aren't we :) ?
 
> your request was very logical.

Well... to tell it all, I reported about such a behavior here and
there, but then, I didn't really pay attention to it... until I was
forced to setup a script, scheduled at intervals, to "trim" the corpus
and restore it to "normal" and, sincerely, given that ASSP has options
to deal with this, I think ASSP *should* deal with this :) and keep the
corpus balanced

> Why is assp not able to produce a fine corpusnorm/spamdb/HMM, if all
> information is available and the folders are full of files?
> Had a sleepness night. I think I've found a way to fix this.

Now ... you make me feel somewhat guilty !! Sleep is a need and
sincerely, causing a sleepless night isn't exactly something I like to
cause (ok, given that the night went wasted thinking to code )

> After the error folders are processed, a temporary corpusnorm is 
> calculated. The files in the spam and notspam folder are counted -
> and depending on the temp-corpusnorm, the spam-file-count and 
> notspam-file-count, the apx. required count of spam files is
> calcuated. If these spam files are finished processed - based on the
> needed notspam word count - the apx. required count of notspam files
> is calculated.
> 
> So (I hope), even if a machine gets too many or too less spams over a
> time , this logic will be able to ensure a fine corpusnorm.

I see, so, basically, you're saying that the weight reported in the
"rebuild report" isn't correct ?!? Not that it's an issue, I can live
with that but... did I get it right ? (sorry if I didn't but last night
I slept 2 hours +/- [yeah, I know, but I was dealing with some *darn*
UTM issues and had to "protect the innocent"] and today I had to travel
@ a customer site... just got back) If so, then, maybe slightly
changing the rebuild code to emit correct values may be a good idea :)
 

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email 

[Assp-test] no sender email in log => ASSP 1.9.7.5(0.0.02)

[Graziano]

Hello

using 1.9.7.5(0.0.02)

sometime I have a row like this in ASSP log

Sep-11-12 00:21:27 id-34733-03401 66.23.233.107 to: wil...@gl.it Message-Score: 
added 46 for Bayesian Probability: 0.99952, total score for this message is now 
91;

where the sender email is not specified .

In the email header the from is clearly specified for example for the email 
above it was the following

From: "Email Marketing" 

Thank you
Graziano

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: Re: Antwort: strange ASSP behavior

[Grayhat]

> >I see, so, basically, you're saying that the weight reported in the
> "rebuild report" isn't correct ?
 
> No - the values were correctly shown. But ASSP has used all files (up
> to MaxFiles) even it was better to use some less ( from here or
> there) to get a better corpusnorm.

Hmm... I see now, basically with the latest change you added some logic
so that older files (which should otherwise be discarded) are ignored
by the rebuild process... am I right ?

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: Re: Antwort: Re: Antwort: strange ASSP behavior

[Thomas Eckardt]

sorry Fritz,

'autoCorrectCorpus' was there as definition for a wanted corpusnorm - but 
with a different mechanism behind

>Target norm is (a+b)/2 .

is now used in addition to the old cleanup mechanism

I think this is what you mean to get a configurable corpusnorm target.

Thomas




Von:"Fritz Borgstedt" 
An: "ASSP development mailing list" , 

Datum:  11.09.2012 16:17
Betreff:Re: [Assp-test] Antwort: Re:  Antwort:  strange ASSP 
behavior



ASSP development mailing list  schrei
bt:
>Target norm is (a+b)/2 .


That was not.

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: Re: Antwort: strange ASSP behavior

[Thomas Eckardt]

>I see, so, basically, you're saying that the weight reported in the
"rebuild report" isn't correct ?

No - the values were correctly shown. But ASSP has used all files (up to 
MaxFiles) even it was better to use some less ( from here or there) to get 
a better corpusnorm.

Thomas



Von:Grayhat 
An: assp-test@lists.sourceforge.net, 
Datum:  11.09.2012 15:25
Betreff:Re: [Assp-test] Antwort:  strange ASSP behavior




> Andrea,

Hi there, Thomas, we are on the public list, aren't we :) ?
 
> your request was very logical.

Well... to tell it all, I reported about such a behavior here and
there, but then, I didn't really pay attention to it... until I was
forced to setup a script, scheduled at intervals, to "trim" the corpus
and restore it to "normal" and, sincerely, given that ASSP has options
to deal with this, I think ASSP *should* deal with this :) and keep the
corpus balanced

> Why is assp not able to produce a fine corpusnorm/spamdb/HMM, if all
> information is available and the folders are full of files?
> Had a sleepness night. I think I've found a way to fix this.

Now ... you make me feel somewhat guilty !! Sleep is a need and
sincerely, causing a sleepless night isn't exactly something I like to
cause (ok, given that the night went wasted thinking to code )

> After the error folders are processed, a temporary corpusnorm is 
> calculated. The files in the spam and notspam folder are counted -
> and depending on the temp-corpusnorm, the spam-file-count and 
> notspam-file-count, the apx. required count of spam files is
> calcuated. If these spam files are finished processed - based on the
> needed notspam word count - the apx. required count of notspam files
> is calculated.
> 
> So (I hope), even if a machine gets too many or too less spams over a
> time , this logic will be able to ensure a fine corpusnorm.

I see, so, basically, you're saying that the weight reported in the
"rebuild report" isn't correct ?!? Not that it's an issue, I can live
with that but... did I get it right ? (sorry if I didn't but last night
I slept 2 hours +/- [yeah, I know, but I was dealing with some *darn*
UTM issues and had to "protect the innocent"] and today I had to travel
@ a customer site... just got back) If so, then, maybe slightly
changing the rebuild code to emit correct values may be a good idea :)
 

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] fixes in assp 2.2.2 build 12255

[Thomas Eckardt]

Hi all,


fixed in assp 2.2.2 build 12255:

- On some linux platforms we saw a CLIB mistake, which caused an IP 
address error in ASSP.
  Even on a connected IP-socket, assp was unable to get the connected IP 
address from the OS.
  This caused unexpected crashes or at least unexpected behavior of assp. 
If this mistake is 
  detected by assp, the connection is closed by assp and 'delaySameIP' is 
switched to ON - which
  sets an internal limit for the count of same IP's in the same worker.
  The detected CLIB mistake is logged in the maillog.txt. In case you see
 
error: This system is some time unable to detect connected IP addresses - 
check that you use the latest C-library, 
Perl-version and Perl module versions 

and/or

error: unable to detect the connected IP address - ...

  check that you use the latest Perl update for your version, and the 
CLIB/LIBC of your OS is uptodate!
 
 
- If a system got peaks of spam or ham mails, the corpusnorm was no longer 
fine - even if the settings for
  the rebuildspamdb and corpus folder maintenance were not changed. ASSP 
tries now on the fly to keep the 
  corpusnorm beween 0.9 and 1.1 . Notice this will only work, if there are 
enougth files in the corpus folders
  (~500 each at least).
 
  'autoCorrectCorpus'
  ...
  If this value is defined, assp will use the middle value of "a" and "b" 
((a+b)/2) as target corpusnorm and 
  will try to reach this value, using (as many as possible) but only such 
a count of files in the folders 
  spamlog and notspamlog as required!'
 

changed:

- The internal config variable 'delaySameIPorHelo' is changed to 
'delaySameIP'

- On havy loaded systems it was possible that the 'command queue' of the 
MaintThread was growing
  for example, if the ARC plugin was used. If the RebuildThread is idle, 
he will also process entries from this
  'command queue'.

- If a connected host submits a HELO without a host or domain name, the 
host name is internaly replaced by 'localhost',
  but the original HELO is sent to the MTA. If 'DoInvalidFormatHelo' is 
enabled, the message and IP will get a penalty of
  'ihValencePB'.

- The SPAM-SPF-record detection is improved. All 'pass' matches with an 
IPv4-network-mask less than 8 or
  an IPv6-network-mask less than 32 are detected as 'fail'.

- If a SPAM-SPF-record is found in the SPFCache for a domain (0.0.0.0 
domain) - this domain will be treated as 
  'blackListedDomain'


added:

- Until now the SMTP timeout for noprocessing and whitelisted mails was 
set hardcoded to 1200 seconds,
  you can change this timeout now with the internal config variable 
'NpWlTimeOut'
our $NpWlTimeOut = 1200;  # timeout in seconds for noprocessing and 
whitelisted mails

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: Re: Antwort: strange ASSP behavior

[Fritz Borgstedt]

ASSP development mailing list  schrei
bt:
>Target norm is (a+b)/2 .


That was not.

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: strange ASSP behavior

[Grayhat]

> Andrea,

Hi there, Thomas, we are on the public list, aren't we :) ?
 
> your request was very logical.

Well... to tell it all, I reported about such a behavior here and
there, but then, I didn't really pay attention to it... until I was
forced to setup a script, scheduled at intervals, to "trim" the corpus
and restore it to "normal" and, sincerely, given that ASSP has options
to deal with this, I think ASSP *should* deal with this :) and keep the
corpus balanced

> Why is assp not able to produce a fine corpusnorm/spamdb/HMM, if all
> information is available and the folders are full of files?
> Had a sleepness night. I think I've found a way to fix this.

Now ... you make me feel somewhat guilty !! Sleep is a need and
sincerely, causing a sleepless night isn't exactly something I like to
cause (ok, given that the night went wasted thinking to code )

> After the error folders are processed, a temporary corpusnorm is 
> calculated. The files in the spam and notspam folder are counted -
> and depending on the temp-corpusnorm, the spam-file-count and 
> notspam-file-count, the apx. required count of spam files is
> calcuated. If these spam files are finished processed - based on the
> needed notspam word count - the apx. required count of notspam files
> is calculated.
> 
> So (I hope), even if a machine gets too many or too less spams over a
> time , this logic will be able to ensure a fine corpusnorm.

I see, so, basically, you're saying that the weight reported in the
"rebuild report" isn't correct ?!? Not that it's an issue, I can live
with that but... did I get it right ? (sorry if I didn't but last night
I slept 2 hours +/- [yeah, I know, but I was dealing with some *darn*
UTM issues and had to "protect the innocent"] and today I had to travel
@ a customer site... just got back) If so, then, maybe slightly
changing the rebuild code to emit correct values may be a good idea :)
 

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: Re: Antwort: strange ASSP behavior

[Thomas Eckardt]

>May be you should make the corpusnorm configurable like 0.9.

Was and is 'autoCorrectCorpus'. Target norm is (a+b)/2 .




Von:"Fritz Borgstedt" 
An: "ASSP development mailing list" , 

Datum:  11.09.2012 13:44
Betreff:Re: [Assp-test] Antwort:  strange ASSP behavior



May be you should make the corpusnorm configurable like 0.9.
Lower norms after softer in deciding spam/notspams. I run my installat
ions with a norm of 0.6 to get less false positives.

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: strange ASSP behavior

[Fritz Borgstedt]

May be you should make the corpusnorm configurable like 0.9.
Lower norms after softer in deciding spam/notspams. I run my installat
ions with a norm of 0.6 to get less false positives.

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: strange ASSP behavior

[Thomas Eckardt]

Andrea,

your request was very logical. Why is assp not able to produce a fine 
corpusnorm/spamdb/HMM, if all information is available and the folders are 
full of files?
Had a sleepness night.
I think I've found a way to fix this.

After the error folders are processed, a temporary corpusnorm is 
calculated. The files in the spam and notspam folder are counted - and 
depending on the temp-corpusnorm, the spam-file-count and 
notspam-file-count, the apx. required count of spam files is calcuated.
If these spam files are finished processed - based on the needed notspam 
word count - the apx. required count of notspam files is calculated.

So (I hope), even if a machine gets too many or too less spams over a time 
, this logic will be able to ensure a fine corpusnorm.

Sep-11-12 11:25:36 c:/assp/errors/spam
Sep-11-12 11:25:36 File Count:  1,039
Sep-11-12 11:25:36 Processing... errors/spam with 1,039 files
Sep-11-12 11:25:37 ignore and remove files older than Dec-16-09 10:25:36 
in folder errors/spam
Sep-11-12 11:28:50 208 attachment/image entries processed
Sep-11-12 11:28:50 Imported Files:  1,037
Sep-11-12 11:28:50 Finished in 194 second(s)

Sep-11-12 11:28:50 c:/assp/errors/notspam
Sep-11-12 11:28:50 File Count:  553
Sep-11-12 11:28:50 Processing... errors/notspam with 553 files
Sep-11-12 11:28:52 ignore and remove files older than Dec-16-09 10:28:50 
in folder errors/notspam
Sep-11-12 11:30:42 96 attachment/image entries processed
Sep-11-12 11:30:42 Imported Files:  551
Sep-11-12 11:30:42 Finished in 112 second(s)
Sep-11-12 11:30:42 info: corpusnorm after processing errors/spam and 
errors/notspam is spamwords 618/ hamwords 1285508 => 0.864730518985491 

Sep-11-12 11:30:42 info: require 1858 files from folder spam to get a fine 
corpusnorm 

Sep-11-12 11:30:42 c:/assp/spam
Sep-11-12 11:30:42 File Count:  2,149
Sep-11-12 11:30:42 Processing... spam with 1,858 files
Sep-11-12 11:30:44 ignore and remove files older than Aug-25-12 11:30:42 
in folder spam
Sep-11-12 11:35:46 Removed Old: 5
Sep-11-12 11:35:46 36 attachment/image entries processed
Sep-11-12 11:35:46 Imported Files:  1,858
Sep-11-12 11:35:46 Finished in 304 second(s)
Sep-11-12 11:35:46 info: require 617 files from folder notspam to get a 
fine corpusnorm 

Sep-11-12 11:35:46 c:/assp/notspam
Sep-11-12 11:35:46 File Count:  599
Sep-11-12 11:35:46 Processing... notspam with 599 files
Sep-11-12 11:35:46 ignore and remove files older than Jul-06-12 11:35:46 
in folder notspam
Sep-11-12 11:37:02 66 attachment/image entries processed
Sep-11-12 11:37:02 Imported Files:  597
Sep-11-12 11:37:02 Finished in 76 second(s)
 ..

Sep-11-12 11:39:26 Spam Weight:1,522,594
Sep-11-12 11:39:26 Not-Spam Weight:   1,499,397

Sep-11-12 11:39:26 Corpus norm: 1.0155 - (very good - balanced)
Sep-11-12 11:39:26 Corpus confidence:   1.
 

The result of the last rebuild without this logic on the same corpus was

Sep-11-12 04:15:32 Spam Weight: 1,605,461
Sep-11-12 04:15:32 Not-Spam Weight:   1,498,629

Sep-11-12 04:15:32 Corpus norm:  1.0713 - (very good - balanced)
Sep-11-12 04:15:32 Corpus confidence:1.


Thomas






Von:Grayhat 
An: assp-test@lists.sourceforge.net, 
Datum:  10.09.2012 13:14
Betreff:[Assp-test] strange ASSP behavior




I'm running the latest ASSP 2.2.2 build 12248 (Win2k8, ActivePerl,
MSSQL), but I observed the same behavior with previous versions as
well; in short, if I manually "trim" the spam/notspam folders down to
14000 files (or less, but same count for both) and start a rebuild, the
rebuild report tells me that the corpus is ok (balanced) but then, if I
leave ASSP running for (say) a week or so (the box gets quite a bunch
of traffic), the spam folder keeps growing and growing and the corpus
quickly moves to "slight spam heavy" and then "too spam heavy" and the
ASSP "automatic cleanup" doesn't seem to help; the rebuild deletes some
"excess files" but the file count seem to be small if compared to the
amount of files stored; the relevant (or they should) parameters in my
config are:

MaxFiles: 14000

FilesDistribution: 1

MaxAllowedDups: 5

MaxBayesFileAge: 30 10

MaxKeepDeleted: 10

MaxCorrectedDays: 1000

MaxNoBayesFileAge: 15

autoCorrectCorpus: 0.6-1.4-4000-10

now... is this a bug or an expected behaviour ? And if it's expected,
what can I do (e.g. config change) to avoid this issue ?


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
**

[Assp-test] Antwort: Re: ASSPV1 and Perl 5.8

[Thomas Eckardt]

>Do you know the source code, the internals, problems, bugs etc. of the 
Perl core and all Perl modules used?

Yes, as long they are published. Without that knowledge ASSP will not 
work. And the ASSP source includes several fixes and workarounds for Perl 
(version dependend !) and module bugs. 

>Windows is highly managed and developed by hundreds of people.

NO - a hand full are working on the kernel - the same way like Linus is 
doing it.
All  the others are working on code that produces nice windows, clickable 
icons and functions , wonderfull desktop animations, a firewall  which 
are not needed to have a running windows server.

>I was talking about ASSP code, not the Perl core or any standard CPAN 
modules.

The "Standard CPAN modules" are part of the Perl core. Even this , but all 
the others are used by ASSP, must be checked.  ASSP has to handle there 
namespace, has to respect or to change there call stack and functions.

>Windows is not just the kernel

Yes it is - use a core installation of Windows 2008R2 and you'll see it.. 
You are right if you speak about the good old times of windows 2000 and 
2003.

Thomas



Von:
An: , 
Datum:  11.09.2012 10:59
Betreff:Re: [Assp-test] ASSPV1 and Perl 5.8



>Yes, the assp code the source of the Perl core and all used modules are 
much larger than the source of any Windows kernel. 

I was talking about ASSP code, not the Perl core or any standard CPAN 
modules. Do you know the source code, the internals, problems, bugs etc. 
of the Perl core and all Perl modules used? And, do you have the source 
code of the windows kernel or know the complete internals of Windows 
beside the information in books like "Windows Internals". Otherwise, it is 
just a statement you make. Windows is not just the kernel as the Perl core 
alone is not ASSP. Windows is highly managed and developed by hundreds of 
people. The ASSP source code V1 and V2 (not CPAN modules or the Perl core) 
is mainly developed by two developers, despite some people like John Hanna 
who started the project or others who added some code blocks. Don't get me 
wrong, I don't want to criticize that fact. This is not a statement on the 
quality of ASSP. Fritz and you Thomas, have done a great job so far, but 
you could not compare ASSP with Windows. ASSP is a tool, Windows is an 
operating system. That is my very onwn opinion.

>Yes - every three years - at least the server base - and every five years 
the complete SAN and network core-switch hardware! The reason is simple - 
with new hardware you'll get three years maintenance nearly for free. 
Paying three years maintenance results in the same price like new 
hardware. This is the short math of the CIO and the Management accounting.

If your company has the money for that, fine! But there are some small 
business companies out there, which don't have that budget.

Thilo

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 10. September 2012 18:04
An: ASSP development mailing list
Betreff: Re: [Assp-test] ASSPV1 and Perl 5.8

>but do you really want to compare the small ASSP code base with Windows

Yes, the assp code the source of the Perl core and all used modules are 
much larger than the source of any Windows kernel.

>windows 2000 and Perl 5.8

. 

>Do they change hardware and software with every new Windows version?

Yes - every three years - at least the server base - and every five years 
the complete SAN and network core-switch hardware! The reason is simple - 
with new hardware you'll get three years maintenance nearly for free. 
Paying three years maintenance results in the same price like new 
hardware. This is the short math of the CIO and the Management accounting 

Thomas





Von:
An: , 
Datum:  10.09.2012 16:05
Betreff:Re: [Assp-test] Antwort: Re:  ASSPV1 and Perl 5.8



>Fritz, if you don't see it, or you don't know it, or you ignore it - it 
>doesn't mean that it is not happen!

Don't you find this a chaos? The developer of V1 and the developer of V2 
have different opinions on the code of V1??? 

Thomas does not recommend running ASSP V1 on Perl below 5.10, Fritz does 
not see any problem with that combination and runs ASSP V1 on Perl 5.8.8 
on production environments, as I do. Now, who is right?

Nevertheless, Thomas thank you for your explanations on problems with 
Unicode character processing.

>No - it would take too long. What was the answer from Microsoft about
your question "what has been changed from Windows >2000 to Windows
2008R2SP1 - and why was it changed?"

I did not even try, maybe I should. But one short comment on this
statement:

Thomas, I know that your are a experienced developer (as I am, but not 
with Perl), but do you really want to compare the small ASSP code base 
with Windows, were hundreds of developers produce code for every new 
version thrown out to public?? Sorry, but ASSP V1 or V2 is a very very 
small piece of 

Re: [Assp-test] ASSPV1 and Perl 5.8

[Thilo.Klein]

>Yes, the assp code the source of the Perl core and all used modules are much 
>larger than the source of any Windows kernel. 

I was talking about ASSP code, not the Perl core or any standard CPAN modules. 
Do you know the source code, the internals, problems, bugs etc. of the Perl 
core and all Perl modules used? And, do you have the source code of the windows 
kernel or know the complete internals of Windows beside the information in 
books like "Windows Internals". Otherwise, it is just a statement you make. 
Windows is not just the kernel as the Perl core alone is not ASSP. Windows is 
highly managed and developed by hundreds of people. The ASSP source code V1 and 
V2 (not CPAN modules or the Perl core) is mainly developed by two developers, 
despite some people like John Hanna who started the project or others who added 
some code blocks. Don't get me wrong, I don't want to criticize that fact. This 
is not a statement on the quality of ASSP. Fritz and you Thomas, have done a 
great job so far, but you could not compare ASSP with Windows. ASSP is a tool, 
Windows is an operating system. That is my very onwn opinion.

>Yes - every three years - at least the server base - and every five years the 
>complete SAN and network core-switch hardware! The reason is simple - with new 
>hardware you'll get three years maintenance nearly for free. 
Paying three years maintenance results in the same price like new hardware. 
This is the short math of the CIO and the Management accounting.

If your company has the money for that, fine! But there are some small business 
companies out there, which don't have that budget.

Thilo

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Gesendet: Montag, 10. September 2012 18:04
An: ASSP development mailing list
Betreff: Re: [Assp-test] ASSPV1 and Perl 5.8

>but do you really want to compare the small ASSP code base with Windows

Yes, the assp code the source of the Perl core and all used modules are much 
larger than the source of any Windows kernel.

>windows 2000 and Perl 5.8

. 

>Do they change hardware and software with every new Windows version?

Yes - every three years - at least the server base - and every five years the 
complete SAN and network core-switch hardware! The reason is simple - with new 
hardware you'll get three years maintenance nearly for free. 
Paying three years maintenance results in the same price like new hardware. 
This is the short math of the CIO and the Management accounting 

Thomas





Von:
An: , 
Datum:  10.09.2012 16:05
Betreff:Re: [Assp-test] Antwort: Re:  ASSPV1 and Perl 5.8



>Fritz, if you don't see it, or you don't know it, or you ignore it - it  
>doesn't mean that it is not happen!

Don't you find this a chaos? The developer of V1 and the developer of V2 have 
different opinions on the code of V1??? 

Thomas does not recommend running ASSP V1 on Perl below 5.10, Fritz does not 
see any problem with that combination and runs ASSP V1 on Perl 5.8.8 on 
production environments, as I do. Now, who is right?

Nevertheless, Thomas thank you for your explanations on problems with Unicode 
character processing.

>No - it would take too long. What was the answer from Microsoft about
your question "what has been changed from Windows >2000 to Windows
2008R2SP1 - and why was it changed?"

I did not even try, maybe I should. But one short comment on this
statement:

Thomas, I know that your are a experienced developer (as I am, but not with 
Perl), but do you really want to compare the small ASSP code base with Windows, 
were hundreds of developers produce code for every new version thrown out to 
public?? Sorry, but ASSP V1 or V2 is a very very small piece of software 
compared with Windows. But this is not a statement on quality.

5.12.4 on Windows 2000 Server

>Are we in 2012 ???

Why change a running system, which does it's work since years on a hardware 
from 2000? Are you working for a company? Do they change hardware and software 
with every new Windows version? I know that Windows 2000 is out of its life 
cycle. The system is on schedule for upgrade. ASSP is a tool for keeping the 
idiots outside and ASSP on Windows 2000, despite some small problems, exactly 
does this. It would be better ASSP would not even be required, do you agree? 
ASSP, as good and free it is, is not a productive software. It really is only 
overhead, but a very necessary overhead.

Thank you both for spending your time on ASSP.

Thilo

-Ursprüngliche Nachricht-
Von: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Gesendet: Samstag, 8. September 2012 10:47
An: ASSP development mailing list
Betreff: [Assp-test] Antwort: Re: ASSPV1 and Perl 5.8

>Tell me exactly were are the conflicts.

No - it would take too long. What was the answer from Microsoft about your 
question "what has been changed from Windows 2000 to Windows 2008R2SP1 - and 
why was it changed?"

short very simple example : s/\S+//