Re: Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Rajesh M]

eric

i did not yet simulate it using your script

these are errors in my live production server during peak hours today and there 
were mails with and without attachments.

the max processes i got was 47 only.

rajesh


- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To: qmailtoaster-list@qmailtoaster.com
Sent: Mon, 4 Sep 2017 11:19:35 -0600
Subject:

Did you hit it with email containing attachments as well?


On 9/4/2017 10:58 AM, Rajesh M wrote:
> eric
>
> the max i saw was 47 connections using the shell script you gave me.
>
> there were a few errors related to qq soft reject but they got delivered.
>
> rajesh
>
>
>
> - Original Message -
> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
> To: qmailtoaster-list@qmailtoaster.com
> Sent: Sat, 2 Sep 2017 12:35:14 -0600
> Subject:
>
> Rajesh,
>
> I'm still not sure the process limit is being reached.
>
> I'm testing a COS6/QMT server now and haven't been able to bring about
> 'qq soft reject' or failure of any sort. I'm hitting it with email using
> a delivery script from two servers with attachment size of 320KB.
>
> try the script below (psmem) to monitor:
>
> 
>
> #!/bin/bash
> ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print
> "Number of processes =",count; print "Memory usage per process
> =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'
>
> 
>
> # chmod 755 psmem
>
> # watch -n 1 ./psmem simscan
>
>
> Not sure I should change anything until we figure out what's going on.
>
> Eric
>
>
>
> On 9/2/2017 12:03 PM, Rajesh M wrote:
>> eric
>>
>> i could not get the bash script working with either of the two options. 
>> still gives error
>>line 23: `return-limits': not a valid identifier.
>>
>>
>>
>> can the limits issue in simscan be fixed ?
>>
>> thanks,
>> rajesh
>>
>> - Original Message -
>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>> To:qmailtoaster-list@qmailtoaster.com
>> Sent: Fri, 1 Sep 2017 23:46:15 -0600
>> Subject:
>>
>> Rajesh,
>>
>> It has something to do with #!/bin/bash or #!/bin/sh.
>>
>> At top of script, when I use #!/bin/sh I get the same error you do, but
>> when I use #!/bin/bash script works fine.
>>
>>
>> Eric
>>
>>
>> On 9/1/2017 11:07 PM, Rajesh M wrote:
>>> eric
>>>
>>> 1) is their any resolution for this issue ?
>>>
>>> 2) if i run the script it throws error as such
>>> .
>>> /limits.sh: line 23: `return-limits': not a valid identifier
>>>
>>> #!/bin/bash
>>> if [ "$#" -ne "1" ]; then
>>>   echo ""
>>>   echo -e "\033[01;32mLimit checker\033[00m"
>>>   echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
>>>   echo ""
>>>   exit 0
>>> fi
>>>
>>> return-limits(){
>>>   for process in $@; do
>>>  process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
>>>
>>>  if [ -z $@ ]; then
>>> echo "[no $process running]"
>>>  else
>>> for pid in $process_pids; do
>>>echo "[$process #$pid -- limits]"
>>>cat /proc/$pid/limits
>>>  done
>>>  fi
>>>   done
>>> }
>>> return-limits $1
>>>
>>> rajesh
>>>
>>>
>>>
>>> - Original Message -
>>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>>> To:qmailtoaster-list@qmailtoaster.com
>>> Sent: Fri, 1 Sep 2017 17:28:40 -0600
>>> Subject:
>>>
>>> I sent myself a large file so that I could examine simscan with the
>>> below script (limits.sh). Here's the output:
>>>
>>> [root@pet105 simscan-1.4.0]# ./limits simscan
>>> [simscan #31535 -- limits]
>>> Limit   
>>>    
>>>    
>>>    
>>>    
>>>    
>>>    Soft Limit 
>>>    
>>>    
>>>     Hard 
>>> Limit   
>>>    
>>>    
>>>   Units
>>> Max cpu time   
>>>    
>>>    
>>>    
>>>   unlimited unlimited 
>>>    
>>>    
>>>   

Re: Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Rajesh M]

eric

i got 3 types of errors like earlier.

failure to fork, ripmime
and  /var/qmail/simscan/1simscan: check_spam had an error ret: -1

what exactly would be  /var/qmail/simscan/1simscan ... there is no such file

do want me to send you a few of my log files directly to your email id ?

rajesh

- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To: qmailtoaster-list@qmailtoaster.com
Sent: Mon, 4 Sep 2017 11:19:04 -0600
Subject:

What kind of qq soft reject errors were there? Failure to fork or ripmime?


On 9/4/2017 10:58 AM, Rajesh M wrote:
> eric
>
> the max i saw was 47 connections using the shell script you gave me.
>
> there were a few errors related to qq soft reject but they got delivered.
>
> rajesh
>
>
>
> - Original Message -
> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
> To: qmailtoaster-list@qmailtoaster.com
> Sent: Sat, 2 Sep 2017 12:35:14 -0600
> Subject:
>
> Rajesh,
>
> I'm still not sure the process limit is being reached.
>
> I'm testing a COS6/QMT server now and haven't been able to bring about
> 'qq soft reject' or failure of any sort. I'm hitting it with email using
> a delivery script from two servers with attachment size of 320KB.
>
> try the script below (psmem) to monitor:
>
> 
>
> #!/bin/bash
> ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print
> "Number of processes =",count; print "Memory usage per process
> =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'
>
> 
>
> # chmod 755 psmem
>
> # watch -n 1 ./psmem simscan
>
>
> Not sure I should change anything until we figure out what's going on.
>
> Eric
>
>
>
> On 9/2/2017 12:03 PM, Rajesh M wrote:
>> eric
>>
>> i could not get the bash script working with either of the two options. 
>> still gives error
>>line 23: `return-limits': not a valid identifier.
>>
>>
>>
>> can the limits issue in simscan be fixed ?
>>
>> thanks,
>> rajesh
>>
>> - Original Message -
>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>> To:qmailtoaster-list@qmailtoaster.com
>> Sent: Fri, 1 Sep 2017 23:46:15 -0600
>> Subject:
>>
>> Rajesh,
>>
>> It has something to do with #!/bin/bash or #!/bin/sh.
>>
>> At top of script, when I use #!/bin/sh I get the same error you do, but
>> when I use #!/bin/bash script works fine.
>>
>>
>> Eric
>>
>>
>> On 9/1/2017 11:07 PM, Rajesh M wrote:
>>> eric
>>>
>>> 1) is their any resolution for this issue ?
>>>
>>> 2) if i run the script it throws error as such
>>> .
>>> /limits.sh: line 23: `return-limits': not a valid identifier
>>>
>>> #!/bin/bash
>>> if [ "$#" -ne "1" ]; then
>>>   echo ""
>>>   echo -e "\033[01;32mLimit checker\033[00m"
>>>   echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
>>>   echo ""
>>>   exit 0
>>> fi
>>>
>>> return-limits(){
>>>   for process in $@; do
>>>  process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
>>>
>>>  if [ -z $@ ]; then
>>> echo "[no $process running]"
>>>  else
>>> for pid in $process_pids; do
>>>echo "[$process #$pid -- limits]"
>>>cat /proc/$pid/limits
>>>  done
>>>  fi
>>>   done
>>> }
>>> return-limits $1
>>>
>>> rajesh
>>>
>>>
>>>
>>> - Original Message -
>>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>>> To:qmailtoaster-list@qmailtoaster.com
>>> Sent: Fri, 1 Sep 2017 17:28:40 -0600
>>> Subject:
>>>
>>> I sent myself a large file so that I could examine simscan with the
>>> below script (limits.sh). Here's the output:
>>>
>>> [root@pet105 simscan-1.4.0]# ./limits simscan
>>> [simscan #31535 -- limits]
>>> Limit   
>>>    
>>>    
>>>    
>>>    
>>>    
>>>    Soft Limit 
>>>    
>>>    
>>>     Hard 
>>> Limit   
>>>    
>>>    
>>>   Units
>>> Max cpu time   
>>>    
>>>    
>>>    
>>>   unlimited unlimited 
>>>    
>>>  

RE: Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Rajesh M]

eric

the max i saw was 47 connections using the shell script you gave me.

there were a few errors related to qq soft reject but they got delivered.

rajesh



- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To: qmailtoaster-list@qmailtoaster.com
Sent: Sat, 2 Sep 2017 12:35:14 -0600
Subject:

Rajesh,

I'm still not sure the process limit is being reached.

I'm testing a COS6/QMT server now and haven't been able to bring about
'qq soft reject' or failure of any sort. I'm hitting it with email using
a delivery script from two servers with attachment size of 320KB.

try the script below (psmem) to monitor:



#!/bin/bash
ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print
"Number of processes =",count; print "Memory usage per process
=",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'



# chmod 755 psmem

# watch -n 1 ./psmem simscan


Not sure I should change anything until we figure out what's going on.

Eric



On 9/2/2017 12:03 PM, Rajesh M wrote:
> eric
>
> i could not get the bash script working with either of the two options. still 
> gives error
>   line 23: `return-limits': not a valid identifier.
>
>
>
> can the limits issue in simscan be fixed ?
>
> thanks,
> rajesh
>
> - Original Message -
> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
> To:qmailtoaster-list@qmailtoaster.com
> Sent: Fri, 1 Sep 2017 23:46:15 -0600
> Subject:
>
> Rajesh,
>
> It has something to do with #!/bin/bash or #!/bin/sh.
>
> At top of script, when I use #!/bin/sh I get the same error you do, but
> when I use #!/bin/bash script works fine.
>
>
> Eric
>
>
> On 9/1/2017 11:07 PM, Rajesh M wrote:
>> eric
>>
>> 1) is their any resolution for this issue ?
>>
>> 2) if i run the script it throws error as such
>> .
>> /limits.sh: line 23: `return-limits': not a valid identifier
>>
>> #!/bin/bash
>> if [ "$#" -ne "1" ]; then
>>  echo ""
>>  echo -e "\033[01;32mLimit checker\033[00m"
>>  echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
>>  echo ""
>>  exit 0
>> fi
>>
>> return-limits(){
>>  for process in $@; do
>> process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
>>
>> if [ -z $@ ]; then
>>echo "[no $process running]"
>> else
>>for pid in $process_pids; do
>>   echo "[$process #$pid -- limits]"
>>   cat /proc/$pid/limits
>> done
>> fi
>>  done
>> }
>> return-limits $1
>>
>> rajesh
>>
>>
>>
>> - Original Message -
>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>> To:qmailtoaster-list@qmailtoaster.com
>> Sent: Fri, 1 Sep 2017 17:28:40 -0600
>> Subject:
>>
>> I sent myself a large file so that I could examine simscan with the
>> below script (limits.sh). Here's the output:
>>
>> [root@pet105 simscan-1.4.0]# ./limits simscan
>> [simscan #31535 -- limits]
>> Limit        
>>         
>>      Soft Limit   
>>         Hard 
>> Limit        
>>    Units
>> Max cpu time       
>>        unlimited 
>> unlimited       
>>      seconds
>> Max file size       
>>       unlimited unlimited 
>>         
>>    bytes
>> Max data size       
>>       unlimited unlimited 
>>         
>>    bytes
>> Max stack size 10485760 unlimited    
>>         bytes
>> Max core file size      
>>   0 unlimited      
>>       bytes
>> Max resident set      
>>     unlimited unlimited   
>>          
>> bytes
>> Max processes 1024 31121     
>>         
>>    processes
>> Max open files       
>>  

Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Eric Broch]

They're all temporary, for scanning purposes. Once the scanning is done, 
they all are deleted.



On 9/3/2017 3:57 PM, Remo Mattei wrote:

I do not see anything on my  :)  /var/qmail/simscan


Remo
On 9/3/17 12:13 PM, Eric Broch wrote:


Rajesh,


Those shouldn't be log files in that directory, but directories of 
this form...


[root@mail ~]# ls -l /var/qmail/simscan
total 7
drwxr-x--- 2 clamav vchkpw 4096 Apr 12  2016 1460525628.43125.19212
drwxr-x--- 2 clamav vchkpw 4096 Apr 12  2016 1460525917.767516.19248
drwxr-x--- 2 clamav vchkpw  131 Apr 12  2016 1460525934.725762.19262
drwxr-x--- 2 clamav vchkpw  131 Apr 12  2016 1460525934.728000.19263
drwxr-x--- 2 clamav vchkpw  131 Apr 12  2016 1460525934.730011.19264
drwxr-x--- 2 clamav vchkpw 4096 Apr 12  2016 1460525959.679164.19279
drwxr-x--- 2 clamav vchkpw  131 Apr 12  2016 1460525972.624759.19287

in which messages are broken into their respective parts (below)...

/var/qmail/simscan/1493074576.527224.5829:
total 1472

-rw-r- 1 clamav vchkpw 37 Apr 24 16:56 
addr.1493074576.527224.5829

-rw-r- 1 clamav vchkpw  20719 Apr 24 16:56 __If_
-rw-r- 1 clamav vchkpw 846597 Apr 24 16:56 msg.1493074576.527224.5829
-rw-r- 1 clamav vchkpw  0 Apr 24 16:56 textfile0
-rw--- 1 clamav vchkpw   9090 Apr 24 16:56 textfile1
-rw--- 1 clamav vchkpw  47495 Apr 24 16:56 textfile2
-rw--- 1 clamav vchkpw 362496 Apr 24 16:56 WGS-VRV Indoor Units.doc
-rw--- 1 clamav vchkpw 203264 Apr 24 16:56 WGS-VRV IV 208-230V 
Heat Pump.doc


for examination.

There should be no log files in /var/qmail/simscan and their should 
be no old files there either.


Eric


On 9/3/2017 12:54 PM, Rajesh M wrote:

eric

around 1 gb spare.

there are a lot of log files.

rajesh

- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To:qmailtoaster-list@qmailtoaster.com
Sent: Sun, 3 Sep 2017 11:21:24 -0600
Subject:

Rajesh,

How much  available space is on the /var/qmail/simscan directory?

Eric
On 9/3/2017 12:19 AM, Rajesh M wrote:

eric

i would like to conduct more load intensive tests.

is it possible to provide me the shell script that you use for this purpose.

rajesh

-
To unsubscribe, e-mail:qmailtoaster-list-unsubscr...@qmailtoaster.com
For additional commands, e-mail:qmailtoaster-list-h...@qmailtoaster.com




-
To unsubscribe, e-mail:qmailtoaster-list-unsubscr...@qmailtoaster.com
For additional commands, e-mail:qmailtoaster-list-h...@qmailtoaster.com






--
Eric Broch
White Horse Technical Consulting (WHTC)

RE: Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Rajesh M]

eric

thanks for the info. the script you sent works

the server processes roughly around 8 emails per day during week days and 
around 80 percent of it comes during around 12 hours. So during peak hours it 
is around 8000 emails per hour.

i will test this out monday morning peak hours and revert.

thanks
rajesh

- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To: qmailtoaster-list@qmailtoaster.com
Sent: Sat, 2 Sep 2017 12:35:14 -0600
Subject:

Rajesh,

I'm still not sure the process limit is being reached.

I'm testing a COS6/QMT server now and haven't been able to bring about
'qq soft reject' or failure of any sort. I'm hitting it with email using
a delivery script from two servers with attachment size of 320KB.

try the script below (psmem) to monitor:



#!/bin/bash
ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print
"Number of processes =",count; print "Memory usage per process
=",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'



# chmod 755 psmem

# watch -n 1 ./psmem simscan


Not sure I should change anything until we figure out what's going on.

Eric



On 9/2/2017 12:03 PM, Rajesh M wrote:
> eric
>
> i could not get the bash script working with either of the two options. still 
> gives error
>   line 23: `return-limits': not a valid identifier.
>
>
>
> can the limits issue in simscan be fixed ?
>
> thanks,
> rajesh
>
> - Original Message -
> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
> To:qmailtoaster-list@qmailtoaster.com
> Sent: Fri, 1 Sep 2017 23:46:15 -0600
> Subject:
>
> Rajesh,
>
> It has something to do with #!/bin/bash or #!/bin/sh.
>
> At top of script, when I use #!/bin/sh I get the same error you do, but
> when I use #!/bin/bash script works fine.
>
>
> Eric
>
>
> On 9/1/2017 11:07 PM, Rajesh M wrote:
>> eric
>>
>> 1) is their any resolution for this issue ?
>>
>> 2) if i run the script it throws error as such
>> .
>> /limits.sh: line 23: `return-limits': not a valid identifier
>>
>> #!/bin/bash
>> if [ "$#" -ne "1" ]; then
>>  echo ""
>>  echo -e "\033[01;32mLimit checker\033[00m"
>>  echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
>>  echo ""
>>  exit 0
>> fi
>>
>> return-limits(){
>>  for process in $@; do
>> process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`
>>
>> if [ -z $@ ]; then
>>echo "[no $process running]"
>> else
>>for pid in $process_pids; do
>>   echo "[$process #$pid -- limits]"
>>   cat /proc/$pid/limits
>> done
>> fi
>>  done
>> }
>> return-limits $1
>>
>> rajesh
>>
>>
>>
>> - Original Message -
>> From: Eric Broch [mailto:ebr...@whitehorsetc.com]
>> To:qmailtoaster-list@qmailtoaster.com
>> Sent: Fri, 1 Sep 2017 17:28:40 -0600
>> Subject:
>>
>> I sent myself a large file so that I could examine simscan with the
>> below script (limits.sh). Here's the output:
>>
>> [root@pet105 simscan-1.4.0]# ./limits simscan
>> [simscan #31535 -- limits]
>> Limit        
>>         
>>      Soft Limit   
>>         Hard 
>> Limit        
>>    Units
>> Max cpu time       
>>        unlimited 
>> unlimited       
>>      seconds
>> Max file size       
>>       unlimited unlimited 
>>         
>>    bytes
>> Max data size       
>>       unlimited unlimited 
>>         
>>    bytes
>> Max stack size 10485760 unlimited    
>>         bytes
>> Max core file size      
>>   0 unlimited      
>>       bytes
>> Max resident set      
>>     unlimited unlimited   
>>          
>> bytes
>> Max processes 1024 31121     
>>     

Fwd: Re: [qmailtoaster] qq soft reject errors on high load

[Eric Broch]

Rajesh,

I'm still not sure the process limit is being reached.

I'm testing a COS6/QMT server now and haven't been able to bring about 
'qq soft reject' or failure of any sort. I'm hitting it with email using 
a delivery script from two servers with attachment size of 320KB.


try the script below (psmem) to monitor:



#!/bin/bash
ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print 
"Number of processes =",count; print "Memory usage per process 
=",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'




# chmod 755 psmem

# watch -n 1 ./psmem simscan


Not sure I should change anything until we figure out what's going on.

Eric



On 9/2/2017 12:03 PM, Rajesh M wrote:

eric

i could not get the bash script working with either of the two options. still 
gives error
  line 23: `return-limits': not a valid identifier.



can the limits issue in simscan be fixed ?

thanks,
rajesh

- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To:qmailtoaster-list@qmailtoaster.com
Sent: Fri, 1 Sep 2017 23:46:15 -0600
Subject:

Rajesh,

It has something to do with #!/bin/bash or #!/bin/sh.

At top of script, when I use #!/bin/sh I get the same error you do, but
when I use #!/bin/bash script works fine.


Eric


On 9/1/2017 11:07 PM, Rajesh M wrote:

eric

1) is their any resolution for this issue ?

2) if i run the script it throws error as such
.
/limits.sh: line 23: `return-limits': not a valid identifier

#!/bin/bash
if [ "$#" -ne "1" ]; then
 echo ""
 echo -e "\033[01;32mLimit checker\033[00m"
 echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
 echo ""
 exit 0
fi

return-limits(){
 for process in $@; do
process_pids=`ps -C $process -o pid --no-headers | cut -d " " -f 2`

if [ -z $@ ]; then
   echo "[no $process running]"
else
   for pid in $process_pids; do
  echo "[$process #$pid -- limits]"
  cat /proc/$pid/limits
done
fi
 done
}
return-limits $1

rajesh



- Original Message -
From: Eric Broch [mailto:ebr...@whitehorsetc.com]
To:qmailtoaster-list@qmailtoaster.com
Sent: Fri, 1 Sep 2017 17:28:40 -0600
Subject:

I sent myself a large file so that I could examine simscan with the
below script (limits.sh). Here's the output:

[root@pet105 simscan-1.4.0]# ./limits simscan
[simscan #31535 -- limits]
Limit                  
   Soft Limit           Hard Limit  
         Units
Max cpu time              unlimited 
unlimited            seconds
Max file size             unlimited 
unlimited            bytes
Max data size             unlimited 
unlimited            bytes
Max stack size 10485760 unlimited            
bytes
Max core file size        0 unlimited     
       bytes
Max resident set          unlimited unlimited  
          bytes
Max processes 1024 31121              
  processes
Max open files            1024 4096   
              files
Max locked memory         65536 65536    
            bytes
Max address space         unlimited unlimited  
          bytes
Max file locks            unlimited 
unlimited            locks
Max pending signals       31121 31121      
          signals
Max msgqueue size         819200 819200    
           bytes
Max nice priority         0       
             0
Max realtime priority     0          
          0
Max realtime timeout      unlimited unlimited     
       us

Notice the 'Max processes' and 'Max stack size', this could very well be
the issue for our simscan failures.

Eric

On 9/1/2017 5:20 PM, Eric Broch wrote:

This looks like a decent script:


#!/bin/bash
if [ "$#" -ne "1" ]; then
echo ""
echo -e "\033[01;32mLimit checker\033[00m"
echo -e "\033[01;37mUsage:\033[01;33m $0 process_name\033[00m"
echo ""
exit 0
fi

return-limits(){
for process in $@; do