Thanks, I hadn't noticed that the default for that was off so I've changed
that too.
Hopefully that means I can stop bugging you now. Thank you for all the help.
On Thu, Oct 6, 2016 at 11:14 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> these line are logged, if 'ConnectionLog' is enabled.
>
> Thomas
>
>
>
>
> Von: cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum: 06.10.2016 11:47
> Betreff: Re: [Assp-test] unable to detect any running worker
>
>
>
> WorkerLog is reduced now thanks.
>
> Re the logging, something further in the file is changing this.
>
> If I run grep against the file for something simple like the character 1
> it
> outputs hits all the way down to:
>
> 2016-10-06 06:19:17 [Worker_1] Worker_1 wakes up
>
> It then stops and outputs "binary file matches" as it sees characters in
> the stream that aren't consistent with the file type.
>
> Are "bad line-end sequence" and "unsupported_SMTP_line_terminator" things
> produced by the higher WorkerLog setting and will they go away now it is
> set back to standard or do I have to change something else to suppress
> them?
>
>
> On Thu, Oct 6, 2016 at 9:50 AM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > >I'm wondering if the Ubuntu implementation just starts at the number 70
> > for some reason.
> >
> > This is not possible. Global filehandle numbers are starting at 0, which
> > is STDOUT on every OS.
> > Don't care about as long as the process runs well.
> >
> > >I believe this is caused by ASSP outputting some raw SMTP data to the
> > logfile.
> >
> > maillog.txt is an UTF-8 file (contains the UTF-8 BOM at its start).
> >
> > If there everyting is running well, reduce the WorkerLog setting.
> >
> > Thomas
> >
> >
> >
> > Von: cw <colin.war...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum: 06.10.2016 10:38
> > Betreff: Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Hi Thomas,
> >
> > Thank you very much for your help and suggestions with this.
> >
> > So far both servers are running without shut down since I put 79_1 on
> > yesterday.
> >
> > I am still seeing lots of can't interrupt worker - for example at
> 04:48:35
> > when the server wasn't doing much I saw 70 lines logged within the same
> > second. One server had a total of 3,500 logs between midnight and 9am.
> >
> > I'm also seeing issues with grep treating the maillog files as binary
> > which
> > I haven't seen before build 16270.
> >
> > I believe this is caused by ASSP outputting some raw SMTP data to the
> > logfile. For example I see a load of none printable characters
> immediately
> > before the timestamp on a line that says 2016-10-06 07:01:16 [Worker_2]
> > [unsupported_SMTP_line_terminator]
> >
> > Similarly I also see lines such as:
> >
> > 2016-10-06 07:01:29 [Worker_2] 5.10.252.206 info: bad line-end sequence
> in
> > ###
> >
> > Where ### is another load of none-printable characters.
> >
> > I presume this is someone running an exploit scanner as the IP address
> > traces to Azerbaijan.
> >
> > All my logging is to standard at the moment except WorkerLog, is there
> any
> > way to have this information limited to the connection debug files so it
> > doesn't make grep treat the maillog as a binary file?
> >
> > I had a think on the number of perl file handles. I can't explain it as
> > all
> > these servers do is run ASSP and exim4. I have a couple of nightly shell
> > scripts but nothing that would run for more than a few seconds. ps only
> > ever shows the ASSP perl process. I'm wondering if the Ubuntu
> > implementation just starts at the number 70 for some reason.
> >
> > On Wed, Oct 5, 2016 at 5:52 PM, Thomas Eckardt
> > <thomas.ecka...@thockar.com>
> > wrote:
> >
> > > 2.5.4 16279_1 is available at CVS /test.
> > >
> > > I tried to optimize the worker lookup process.
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > >
> > > Von: cw <colin.war...@gmail.com>
> > > An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> > > Datum: 05.10.2016 18:08
> > > Betreff: Re: [Assp-test] unable to detect any running worker
> > >
> > >
> > >
> > > So actually a development.
> > >
> > > It looks like disabling EnableHighPerformance has lightened the load a
> > > little. I've gone an hour without a complete failure and it explains
> why
> > > I'm now seeing "Main_Thread can't interrupt Worker" instead of "unable
> > to
> > > detect any running worker".
> > >
> > > Presumably with high performance set to off ASSP has enough time to
> > > recover
> > > when the issue occurs. Hopefully this will buy me some leeway from
> > having
> > > to find an alternative and find the root cause of the issue.
> > >
> > > On Wed, Oct 5, 2016 at 4:40 PM, cw <colin.war...@gmail.com> wrote:
> > >
> > > > Unfortunately not. Most of the time the GUI is so unresponsive that
> I
> > > only
> > > > go into it when necessary. I understand the performance impact on
> it.
> > > >
> > > > Everything works well when ASSP starts, but within a few minutes it
> > all
> > > > starts to slow down and become unresponsive. I have a feeling that
> it
> > is
> > > > either going to be something really simple or some odd combination
> of
> > > > events that I have been unfortunate enough to end up with.
> > > >
> > > > Most of the time I use command line and tail/grep/vi. Incidentally
> > since
> > > > yesterday I'm getting issues with grep reporting binary file matches
> > on
> > > the
> > > > logs instead of outputting the results. I can only presume some of
> the
> > > > extra debug information is fooling grep but it doesn't help trying
> to
> > > track
> > > > down the right lights in the log!
> > > >
> > > > On Wed, Oct 5, 2016 at 4:32 PM, Thomas Eckardt
> > > <thomas.ecka...@thockar.com
> > > > > wrote:
> > > >
> > > >> >2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > > interrupted
> > > >> >Worker_4 in 10.758 seconds - got (ok)
> > > >>
> > > >> I'm looking for the reason of this very huge time value of 10.758
> > > seconds.
> > > >>
> > > >> The only reason I can find is - you are running constantly at least
> > one
> > > >> auto refreshing GUI screen. I'm right?
> > > >>
> > > >> switch to 'tail' the maillog.txt on a console session
> > > >>
> > > >> tail -fn 30 /usr...../maillog.txt
> > > >>
> > > >> do not use the shutdown_list (connection screen)!
> > > >>
> > > >> How it works now?
> > > >>
> > > >> Thomas
> > > >>
> > > >>
> > > >>
> > > >> Von: cw <colin.war...@gmail.com>
> > > >> An: ASSP development mailing list
> > <assp-test@lists.sourceforge.net>
> > > >> Datum: 05.10.2016 17:05
> > > >> Betreff: Re: [Assp-test] unable to detect any running worker
> > > >>
> > > >>
> > > >>
> > > >> I specifically checked netstat for established connections and
> didn't
> > > >> include any in the wait state or part open state. According to the
> OS
> > > >> these
> > > >> are connections still fully open. I excluded the ones to the MTA so
> > > this
> > > >> should be purely inbound SMTP connections.
> > > >>
> > > >> I've set EnableHighPerformance back to off.
> > > >>
> > > >> Loads of runs produced:
> > > >>
> > > >> 0.003068
> > > >> 0.003083
> > > >> 0.003115
> > > >> 0.003108
> > > >> 0.003101
> > > >> 0.003095
> > > >> 0.0031
> > > >> 0.003096
> > > >> 0.003096
> > > >> 0.003096
> > > >> 0.003105
> > > >> 0.003099
> > > >> 0.003097
> > > >> 0.003095
> > > >> 0.00308
> > > >> 0.003071
> > > >> 0.003071
> > > >> 0.003083
> > > >> 0.003077
> > > >> 0.003072
> > > >> 0.003076
> > > >> 0.003075
> > > >> 0.003075
> > > >> 0.003072
> > > >> 0.0031
> > > >> 0.003077
> > > >> 0.003094
> > > >> 0.003099
> > > >>
> > > >>
> > > >> On Wed, Oct 5, 2016 at 3:45 PM, Thomas Eckardt
> > > >> <thomas.ecka...@thockar.com>
> > > >> wrote:
> > > >>
> > > >> > >I'm currently comparing things. shutdown_list shows 5 smtp
> > sessions
> > > in
> > > >> > >threads whilst netstat shows 18 connections established to port
> 25
> > > from
> > > >> > the
> > > >> > >outside world.
> > > >> > >It then went down to 2 SMTP sessions in threads and netstat
> showed
> > > 13
> > > >> > >connections.
> > > >> >
> > > >> > the state of most of the connection should than be something like
> > > >> > "fin_wait". Closed connection are keeped by the OS socket handler
> > for
> > > >> two
> > > >> > minutes to satisfy the peer, if there are still pakets on the
> way.
> > > >> >
> > > >> > set 'EnableHighPerformance' back to the default - 'OFF'
> > > >> >
> > > >> >
> > > >> > please run the following small perl script and tell me the
> output,
> > > >> > Possibly run it multiple times. Is the sleep time very
> different?
> > > >> > I expect to see a sleep time of 0.003 to 0.005 seconds
> > > >> >
> > > >> > use strict;
> > > >> > use Time::HiRes;
> > > >> >
> > > >> > my $t = 3000/1000000;
> > > >> > print Time::HiRes::sleep($t);
> > > >> >
> > > >> > Thomas
> > > >> >
> > > >> >
> > > >> > Von: cw <colin.war...@gmail.com>
> > > >> > An: ASSP development mailing list
> > > <assp-test@lists.sourceforge.net>
> > > >> > Datum: 05.10.2016 16:27
> > > >> > Betreff: Re: [Assp-test] unable to detect any running
> worker
> > > >> >
> > > >> >
> > > >> >
> > > >> > ThreadCycleTime is set to 3000 microseconds
> > > >> > EnableHighPerformance I have changed during the course of this
> week
> > > with
> > > >> > no
> > > >> > effect on the problem. It was at default and is now at the
> highest
> > > >> > setting.
> > > >> > I'd check the exact names for the settings except neither server
> > will
> > > >> give
> > > >> > me a connection to the GUI at the moment.
> > > >> >
> > > >> > I'm currently comparing things. shutdown_list shows 5 smtp
> sessions
> > > in
> > > >> > threads whilst netstat shows 18 connections established to port
> 25
> > > from
> > > >> > the
> > > >> > outside world.
> > > >> > It then went down to 2 SMTP sessions in threads and netstat
> showed
> > 13
> > > >> > connections.
> > > >> >
> > > >> > On Wed, Oct 5, 2016 at 3:06 PM, Thomas Eckardt
> > > >> > <thomas.ecka...@thockar.com>
> > > >> > wrote:
> > > >> >
> > > >> > > What are your setting for :
> > > >> > >
> > > >> > > ThreadCycleTime
> > > >> > > EnableHighPerformance
> > > >> > >
> > > >> > > Thomas
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Von: cw <colin.war...@gmail.com>
> > > >> > > An: ASSP development mailing list
> > > >> <assp-test@lists.sourceforge.net>
> > > >> > > Datum: 05.10.2016 15:51
> > > >> > > Betreff: Re: [Assp-test] unable to detect any running
> > worker
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > I can confirm that this number is artificially high quite
> easily.
> > > >> > > I actually have a third ASSP instance. This one is specially
> > > >> configured
> > > >> > to
> > > >> > > only accept emails from Office 365 and acts as an outbound
> relay
> > > >> purely
> > > >> > to
> > > >> > > gather messages for the corpus. This one does not experience
> the
> > > >> problem
> > > >> > > with no running workers being available.
> > > >> > >
> > > >> > > Its stats haven't been reset for a long time and show an
> average
> > of
> > > >> 2901
> > > >> > > per day. Concurrent SMTP sessions since last restart are 0 (4
> > max).
> > > >> > >
> > > >> > > The last message showing in the logs says:
> > > >> > >
> > > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread got
> > connection
> > > >> > request
> > > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread looks up
> the
> > > best
> > > >> > > Worker for new connection - 75
> > > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread will wait
> > (max
> > > 30
> > > >> s)
> > > >> > > for the answer of Worker_7 which handles 0 sockets
> > > >> > > 2016-10-05 14:43:57 [Worker_7] Worker_7 wakes up
> > > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread freed by
> idle
> > > >> > Worker_7
> > > >> > > in 0.005 seconds - got (ok)
> > > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 got connection
> from
> > > >> > > MainThread - 75/75
> > > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 freed Main_Thread
> -
> > > 84
> > > >> > >
> > > >> > > There are no other emails going on at this time. Just the one
> > > >> connection
> > > >> > > and it still says 75. This server is firewalled and the only
> > > messages
> > > >> it
> > > >> > > ever sees are from known Office 365 IP addresses along with the
> > > >> internal
> > > >> > > test server that sends 1 message every 60 seconds.
> > > >> > >
> > > >> > > On Wed, Oct 5, 2016 at 2:33 PM, cw <colin.war...@gmail.com>
> > wrote:
> > > >> > >
> > > >> > > > Hi Thomas,
> > > >> > > >
> > > >> > > > Thank you for the rundown. The load is exceptionally high at
> > the
> > > >> > moment
> > > >> > > > because everything backs up every time the mail servers stop.
> > > Having
> > > >> > > said
> > > >> > > > that, I've just watched it happen again on one of the
> servers.
> > It
> > > >> only
> > > >> > > had
> > > >> > > > 7 messages showing in the shutdown_list when it happened.
> > Looking
> > > at
> > > >> > the
> > > >> > > > logs I see:
> > > >> > > >
> > > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection
> > from
> > > >> > > > MainThread - 78/78
> > > >> > > > 2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > > >> > interrupted
> > > >> > > > Worker_4 in 10.758 seconds - got (ok)
> > > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed
> Main_Thread
> > -
> > > >> 101
> > > >> > > >
> > > >> > > > So something is artificially inflating the connection count.
> My
> > > best
> > > >> > > guess
> > > >> > > > ties in with something else I was looking at where a remote
> > > server
> > > >> was
> > > >> > > > opening two connections that never got answered. It only got
> > > through
> > > >> > on
> > > >> > > the
> > > >> > > > third connection and that could be replicated each time. I
> put
> > it
> > > >> down
> > > >> > > to
> > > >> > > > some odd firewall on their end but maybe I was wrong in doing
> > > that.
> > > >> > > >
> > > >> > > > The actual stats on one mailserver are:
> > > >> > > >
> > > >> > > > Last reset 8 days, 0 hours, 6 mins
> > > >> > > > Messages process 165413 (20664 per day)
> > > >> > > > SMTP Connections received 113278
> > > >> > > >
> > > >> > > > The other is:
> > > >> > > > Last reset 8 days, 0 hours, 18 mins
> > > >> > > > Messaged processed 131720 (16438 per day)
> > > >> > > > SMTP Connections received 93088
> > > >> > > >
> > > >> > > > The stats reset themselves during on of the upgrades of ASSP,
> > not
> > > >> sure
> > > >> > > why.
> > > >> > > >
> > > >> > > > Each server is 16 CPU cores, 12GB RAM. DB is on a separate
> > MySQL
> > > box
> > > >> > > > though MTA and Clam run local. URIBL is disabled. I don't use
> > > OCR,
> > > >> DCC
> > > >> > > or
> > > >> > > > Razor. I tried to get OCR working once and all hell broke
> lose
> > > and I
> > > >> > > > haven't been back since.
> > > >> > > >
> > > >> > > > I've been monitoring the performance of the servers all along
> > and
> > > >> not
> > > >> > > > seeing any bottlenecks in CPU, RAM or network.When the
> problem
> > > >> started
> > > >> > > the
> > > >> > > > mail servers had 8 and 10GB RAM respectively (10GB for the
> one
> > > >> running
> > > >> > > > rebuild).
> > > >> > > > All the best,
> > > >> > > > Colin
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt
> > > >> > > <thomas.ecka...@thockar.com
> > > >> > > > > wrote:
> > > >> > > >
> > > >> > > >> It seems, that this system is overloaded with active
> > > connections.
> > > >> > > >>
> > > >> > > >> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed
> by
> > > >> > > interrupted
> > > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > > >> > > >>
> > > >> > > >> Even this time value is ten times higher than expected.
> > > >> > > >>
> > > >> > > >> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> > Main_Thread
> > > -
> > > >> > 170
> > > >> > > >>
> > > >> > > >> The '170' is the filehandle number in perl (counted from
> > 1.....)
> > > -
> > > >> > this
> > > >> > > >> implies 70 or more active connections.
> > > >> > > >>
> > > >> > > >> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will
> wait
> > > (max
> > > >> > 30
> > > >> > > s)
> > > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > > >> > > >>
> > > >> > > >> This implies the same, if the equal balancing of all
> > connections
> > > >> over
> > > >> > > all
> > > >> > > >> seven workers works correctly.
> > > >> > > >>
> > > >> > > >> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2
> seconds
> > > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > 0.103252172470093
> > > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > >> 0.0278699398040771
> > > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > >> 0.0423040390014648
> > > >> > > >>
> > > >> > > >> The processing time itself seems to be normal. SC-Time - is
> > the
> > > >> time
> > > >> > > >> required to process one loop (read + queue or read + check +
> > > queue)
> > > >> > for
> > > >> > > >> one active connection.
> > > >> > > >>
> > > >> > > >> Assuming an average of 4 second life time per connection -
> the
> > > >> > workload
> > > >> > > >> per day would be 1.5 million (70 / 4 *3600 *24) connections
> > > based
> > > >> on
> > > >> > > the
> > > >> > > >> provided output.
> > > >> > > >> Configured to an ISP mode the avg. maximum for a single assp
> > > >> instance
> > > >> > > is
> > > >> > > >> 800.000 connections. To be able to handle workload peeks,
> the
> > > >> > > >> configuration should not allow more than 400.000 connection
> a
> > > day.
> > > >> > > >>
> > > >> > > >> ISP mode:
> > > >> > > >> - 16 CPU cores
> > > >> > > >> - 16 Workers
> > > >> > > >> - 24 GB RAM
> > > >> > > >> - dedicated systems for MTA, enterprise database, ClamAV
> > > >> > > >> - HMM and spamDB holded unshared in RAM
> > > >> > > >> - high performance DNS servers
> > > >> > > >> - limited DNS usage (disable most extensive DNS-using checks
> -
> > > eg.
> > > >> > > URIBL)
> > > >> > > >> - no OCR
> > > >> > > >> - no DCC
> > > >> > > >> - no Razor2
> > > >> > > >>
> > > >> > > >> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed
> by
> > > >> > > interrupted
> > > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > > >> > > >>
> > > >> > > >> This time value is near the end of the line. At this time a
> > > small
> > > >> > count
> > > >> > > of
> > > >> > > >> additionally connections can lead in to an assp shutdown,
> > > because
> > > >> the
> > > >> > > >> workers don't accept new connections.
> > > >> > > >>
> > > >> > > >> Thomas
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> Von: cw <colin.war...@gmail.com>
> > > >> > > >> An: ASSP development mailing list
> > > >> > <assp-test@lists.sourceforge.net>
> > > >> > > >> Datum: 05.10.2016 13:15
> > > >> > > >> Betreff: Re: [Assp-test] unable to detect any running
> > > worker
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> Thanks.
> > > >> > > >>
> > > >> > > >> I've had both servers come up against unable to detect any
> > > running
> > > >> > > worker
> > > >> > > >> since clearing out all the files suggested. So I'm getting
> > 16279
> > > >> > > running
> > > >> > > >> now.
> > > >> > > >>
> > > >> > > >> I noticed the startup with those files removed was really
> > quick,
> > > >> > > starting
> > > >> > > >> back up the second time took several minutes so presumably
> > > reading
> > > >> > > those
> > > >> > > >> files during startup takes a little while versus creating
> them
> > > >> fresh.
> > > >> > > >>
> > > >> > > >> I've caught one already and traced it through the new logs:
> > > >> > > >>
> > > >> > > >> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got
> > > connection
> > > >> > > request
> > > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up
> > the
> > > >> best
> > > >> > > >> Worker for new connection - 73
> > > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt
> > worker
> > > >> > > Worker_3
> > > >> > > >> (12) for new connection
> > > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread
> > interrupted
> > > >> > > Worker_3
> > > >> > > >> (12) to submit the connection
> > > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will
> wait
> > > (max
> > > >> 30
> > > >> > > s)
> > > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > > >> > > >> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3:
> > > 0.0462169647216797
> > > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection
> > > from
> > > >> > > >> MainThread - 73/73
> > > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> > Main_Thread
> > > -
> > > >> 170
> > > >> > > >> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
> > > >> > > interrupted
> > > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > > >> > > >> 2016-10-05 11:47:41 [Worker_3] Connected:
> session:7FF3AA6E9A28
> > > >> > > >> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > 0.103252172470093
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > 0.0278699398040771
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply]
> 220
> > > >> > > >> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct
> 2016
> > > >> > 11:47:41
> > > >> > > >> +0100
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > > 0.0423040390014648
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time
> Worker_3:
> > > >> > > >> 0.0296478271484375
> > > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2
> seconds
> > > >> > > >> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
> > > >> > > interrupted
> > > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > > >> > > >>
> > > >> > > >> In this case, it looks like the connection ended without any
> > > actual
> > > >> > > data.
> > > >> > > >> There is lots of activity reported by Worker_3 in between
> > > 11:47:43
> > > >> > and
> > > >> > > >> 11:48:13 but this all pertains to other connections that
> were
> > > >> already
> > > >> > > in
> > > >> > > >> progress.
> > > >> > > >>
> > > >> > > >> That is a bit different to the earlier one where a message
> was
> > > >> > received
> > > >> > > >> and
> > > >> > > >> the message completed within the 30s window.
> > > >> > > >>
> > > >> > > >> I'm not seeing anything to help me figure out why though and
> I
> > > >> don't
> > > >> > > want
> > > >> > > >> to simply post a big excerpt of the maillog.txt.
> > > >> > > >>
> > > >> > > >> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
> > > >> > > >> <thomas.ecka...@thockar.com>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > >I looked at SF but only see 16275 in test (updated 3 days
> > > ago)
> > > >> so
> > > >> > > maybe
> > > >> > > >> > it
> > > >> > > >> > hasn't made its way live yet.
> > > >> > > >> >
> > > >> > > >> > Sorry, my background CVS sync was not running - update is
> > > done.
> > > >> > > >> >
> > > >> > > >> > Thomas
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > Von: cw <colin.war...@gmail.com>
> > > >> > > >> > An: ASSP development mailing list
> > > >> > > <assp-test@lists.sourceforge.net>
> > > >> > > >> > Datum: 05.10.2016 11:48
> > > >> > > >> > Betreff: Re: [Assp-test] unable to detect any
> running
> > > >> worker
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > Thank you Thomas.
> > > >> > > >> >
> > > >> > > >> > useDB4IntCache - already set to off
> > > >> > > >> > I've set WorkerLog to diagnostic and done the other steps.
> > > >> > > >> > I don't have anything in CorrectASSPcfg.pm so as part of
> the
> > > >> > > >> > troubleshooting I have previously deleted it and
> downloaded
> > a
> > > >> fresh
> > > >> > > copy
> > > >> > > >> > from SourceForge.
> > > >> > > >> >
> > > >> > > >> > I have to say that ASSP started up within seconds after
> > > clearing
> > > >> > > those
> > > >> > > >> > files out - it normally takes several minutes and has
> always
> > > >> done.
> > > >> > > >> >
> > > >> > > >> > I looked at SF but only see 16275 in test (updated 3 days
> > ago)
> > > so
> > > >> > > maybe
> > > >> > > >> it
> > > >> > > >> > hasn't made its way live yet.
> > > >> > > >> >
> > > >> > > >> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> > > >> > > >> > <thomas.ecka...@thockar.com>
> > > >> > > >> > wrote:
> > > >> > > >> >
> > > >> > > >> > > I've provided an updated assp.pl (2.5.4 16279) in CVS
> > /test.
> > > >> This
> > > >> > > >> > version
> > > >> > > >> > > shows some more information, if 'WorkerLog' is set to
> > > >> diagnostic.
> > > >> > > >> > >
> > > >> > > >> > > Thomas
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > > Von: cw <colin.war...@gmail.com>
> > > >> > > >> > > An: ASSP development mailing list
> > > >> > > >> <assp-test@lists.sourceforge.net>
> > > >> > > >> > > Datum: 05.10.2016 10:10
> > > >> > > >> > > Betreff: Re: [Assp-test] unable to detect any
> > running
> > > >> > worker
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > > Hi Thomas,
> > > >> > > >> > >
> > > >> > > >> > > Thanks for chipping in. All modules are installed by
> > running
> > > >> the
> > > >> > > >> latest
> > > >> > > >> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in
> > it
> > > so
> > > >> it
> > > >> > > >> > requires
> > > >> > > >> > > a minor edit to the code to get it to build. So that
> > module
> > > was
> > > >> > > >> > installed
> > > >> > > >> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and
> > > >> running:
> > > >> > > >> > > make clean
> > > >> > > >> > > perl Makefile.PL
> > > >> > > >> > > make
> > > >> > > >> > > make test
> > > >> > > >> > > make install
> > > >> > > >> > >
> > > >> > > >> > > I run cpan-outdated -p|cpanm from time to time in order
> to
> > > keep
> > > >> > > >> modules
> > > >> > > >> > up
> > > >> > > >> > > to date as well so things don't stay stuck on old
> > versions.
> > > >> > > >> > >
> > > >> > > >> > > Everything that is installed now is a completely fresh
> > build
> > > as
> > > >> > the
> > > >> > > >> > > Upgrade
> > > >> > > >> > > to 16.04 replaced perl 5.18 with 5.22 and all the
> modules
> > > >> > therefore
> > > >> > > >> had
> > > >> > > >> > to
> > > >> > > >> > > be installed from scratch.
> > > >> > > >> > >
> > > >> > > >> > > I bypassed the issue by truncating the tables and
> setting
> > up
> > > >> the
> > > >> > > users
> > > >> > > >> > > again. With there only being one user it was easier to
> do
> > > that
> > > >> > than
> > > >> > > >> muck
> > > >> > > >> > > about with it - especially with the other bigger issue
> at
> > > hand.
> > > >> > > >> > >
> > > >> > > >> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > > >> > > >> > > <thomas.ecka...@thockar.com>
> > > >> > > >> > > wrote:
> > > >> > > >> > >
> > > >> > > >> > > > >to replace the corrupted encrypted
> > > >> > > >> > > > strings with the correct values
> > > >> > > >> > > >
> > > >> > > >> > > > Was the 'Crypt::GOST' module from the SF download page
> > at
> > > the
> > > >> > old
> > > >> > > >> assp
> > > >> > > >> > > > instance?
> > > >> > > >> > > > If it was, did you install the 'Crypt::GOST' module
> from
> > > the
> > > >> SF
> > > >> > > >> > download
> > > >> > > >> > > > page, before you started the new assp instance?
> > > >> > > >> > > >
> > > >> > > >> > > > https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > > >> > > >> > > >
> > > >> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > > >> > > >> > > >
> > > >> > > >> > > > Thomas
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > > Von: cw <colin.war...@gmail.com>
> > > >> > > >> > > > An: ASSP development mailing list
> > > >> > > >> > <assp-test@lists.sourceforge.net>
> > > >> > > >> > > > Datum: 05.10.2016 09:52
> > > >> > > >> > > > Betreff: Re: [Assp-test] unable to detect any
> > > running
> > > >> > > worker
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > > Cheers for the reply.
> > > >> > > >> > > >
> > > >> > > >> > > > I don't think backups are an option seen as I've moved
> > > >> > completely
> > > >> > > >> from
> > > >> > > >> > > > Ubuntu 14.04 to Ubuntu 16.04.
> > > >> > > >> > > >
> > > >> > > >> > > > Also, this issue has been around for months. It was
> > > causing a
> > > >> > > >> handful
> > > >> > > >> > of
> > > >> > > >> > > > shutdowns a week with the occasional spat of more
> > frequent
> > > >> > > >> shutdowns.
> > > >> > > >> > It
> > > >> > > >> > > > is
> > > >> > > >> > > > entirely possible that the errors are behaviour
> related
> > > and
> > > >> > > nothing
> > > >> > > >> to
> > > >> > > >> > > do
> > > >> > > >> > > > with the upgrade and that current email behaviour is
> > > >> triggering
> > > >> > > it
> > > >> > > >> big
> > > >> > > >> > > > style.
> > > >> > > >> > > >
> > > >> > > >> > > > I'm not convinced though, unfortunately I'm not
> > convinced
> > > of
> > > >> > > >> anything
> > > >> > > >> > > else
> > > >> > > >> > > > hence not having much to go on. I can't see any
> > > consistencies
> > > >> > in
> > > >> > > the
> > > >> > > >> > > > behaviour leading up to the events.
> > > >> > > >> > > >
> > > >> > > >> > > > I don't think it is database related. It happened on
> one
> > > of
> > > >> the
> > > >> > > mail
> > > >> > > >> > > > servers during the upgrade to 16.04 when ASSP had just
> > > >> started
> > > >> > > but I
> > > >> > > >> > had
> > > >> > > >> > > > not yet got into the web interface to replace the
> > > corrupted
> > > >> > > >> encrypted
> > > >> > > >> > > > strings with the correct values so all database
> > > connections
> > > >> > were
> > > >> > > in
> > > >> > > >> > > error.
> > > >> > > >> > > >
> > > >> > > >> > > > The problem has already started again this morning so
> I
> > > can
> > > >> see
> > > >> > > this
> > > >> > > >> > > being
> > > >> > > >> > > > another fun day that either leads to a fix or having
> to
> > > put
> > > >> > > >> something
> > > >> > > >> > > else
> > > >> > > >> > > > in place.
> > > >> > > >> > > >
> > > >> > > >> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post
> > > <nntp.p...@gmail.com>
> > > >> > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > > I've been reading here, but I haven't had anything
> to
> > > >> > suggest.
> > > >> > > All
> > > >> > > >> > > > seems
> > > >> > > >> > > > > quite odd if it was working prior to upgrading and
> > > >> > downgrading
> > > >> > > >> > didn't
> > > >> > > >> > > > work.
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > > Could you spin up a backup of the installation after
> > > >> copying
> > > >> > > the
> > > >> > > >> > > current
> > > >> > > >> > > > > data? Sure you'd have an older corpus, but I'd
> think
> > > you
> > > >> > could
> > > >> > > >> add
> > > >> > > >> > > the
> > > >> > > >> > > > new
> > > >> > > >> > > > > files if necessary, manually replace whitelist etc.
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw
> > > <colin.war...@gmail.com
> > > >> >
> > > >> > > >> wrote:
> > > >> > > >> > > > >
> > > >> > > >> > > > > > Further development on this today, very little.
> > > >> > > >> > > > > > I have moved both servers onto Ubuntu 16.04 LTS
> > which
> > > >> means
> > > >> > > >> going
> > > >> > > >> > > from
> > > >> > > >> > > > > perl
> > > >> > > >> > > > > > 5.18 to 5.22 and rebuilding all perl modules from
> > > >> scratch.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > The admin user db did not work after the upgrade
> so
> > I
> > > had
> > > >> > to
> > > >> > > >> empty
> > > >> > > >> > > the
> > > >> > > >> > > > > > tables before it would come back online.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > I'm still getting delayed emails and assp shutting
> > > down
> > > >> > > telling
> > > >> > > >> me
> > > >> > > >> > > it
> > > >> > > >> > > > is
> > > >> > > >> > > > > > unable
> > > >> > > >> > > > > > to detect any running worker.
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > If this goes on much longer the MD will pull the
> > plug
> > > and
> > > >> > > we'll
> > > >> > > >> > end
> > > >> > > >> > > up
> > > >> > > >> > > > > > moving to a third party solution which is not
> > > something I
> > > >> > > want
> > > >> > > >> but
> > > >> > > >> > > if
> > > >> > > >> > > > I
> > > >> > > >> > > > > > can't fix it I can't defend it :/
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > ------------------------------------------------------------
> > > >> > > >> > > > > > ------------------
> > > >> > > >> > > > > > Check out the vibrant tech community on one of the
> > > >> world's
> > > >> > > most
> > > >> > > >> > > > > > engaging tech sites, SlashDot.org!
> > > >> http://sdm.link/slashdot
> > > >> > > >> > > > > > _______________________________________________
> > > >> > > >> > > > > > Assp-test mailing list
> > > >> > > >> > > > > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > > > >
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> ------------------------------------------------------------
> > > >> > > >> > > > > ------------------
> > > >> > > >> > > > > Check out the vibrant tech community on one of the
> > > world's
> > > >> > most
> > > >> > > >> > > > > engaging tech sites, SlashDot.org!
> > > >> http://sdm.link/slashdot
> > > >> > > >> > > > > _______________________________________________
> > > >> > > >> > > > > Assp-test mailing list
> > > >> > > >> > > > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > > >
> https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > ------------------------------------------------------------
> > > >> > > >> > > > ------------------
> > > >> > > >> > > > Check out the vibrant tech community on one of the
> > world's
> > > >> most
> > > >> > > >> > > > engaging tech sites, SlashDot.org!
> > > http://sdm.link/slashdot
> > > >> > > >> > > > _______________________________________________
> > > >> > > >> > > > Assp-test mailing list
> > > >> > > >> > > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > > DISCLAIMER:
> > > >> > > >> > > >
> *******************************************************
> > > >> > > >> > > > This email and any files transmitted with it may be
> > > >> > confidential,
> > > >> > > >> > > legally
> > > >> > > >> > > > privileged and protected in law and are intended
> solely
> > > for
> > > >> the
> > > >> > > use
> > > >> > > >> of
> > > >> > > >> > > the
> > > >> > > >> > > >
> > > >> > > >> > > > individual to whom it is addressed.
> > > >> > > >> > > > This email was multiple times scanned for viruses.
> There
> > > >> should
> > > >> > > be
> > > >> > > >> no
> > > >> > > >> > > > known virus in this email!
> > > >> > > >> > > >
> *******************************************************
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > ------------------------------------------------------------
> > > >> > > >> > > > ------------------
> > > >> > > >> > > > Check out the vibrant tech community on one of the
> > world's
> > > >> most
> > > >> > > >> > > > engaging tech sites, SlashDot.org!
> > > http://sdm.link/slashdot
> > > >> > > >> > > > _______________________________________________
> > > >> > > >> > > > Assp-test mailing list
> > > >> > > >> > > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > ------------------------------------------------------------
> > > >> > > >> > > ------------------
> > > >> > > >> > > Check out the vibrant tech community on one of the
> world's
> > > most
> > > >> > > >> > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > > >> > > >> > > _______________________________________________
> > > >> > > >> > > Assp-test mailing list
> > > >> > > >> > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > > DISCLAIMER:
> > > >> > > >> > > *******************************************************
> > > >> > > >> > > This email and any files transmitted with it may be
> > > >> confidential,
> > > >> > > >> > legally
> > > >> > > >> > > privileged and protected in law and are intended solely
> > for
> > > the
> > > >> > use
> > > >> > > of
> > > >> > > >> > the
> > > >> > > >> > >
> > > >> > > >> > > individual to whom it is addressed.
> > > >> > > >> > > This email was multiple times scanned for viruses. There
> > > should
> > > >> > be
> > > >> > > no
> > > >> > > >> > > known virus in this email!
> > > >> > > >> > > *******************************************************
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > >
> > ------------------------------------------------------------
> > > >> > > >> > > ------------------
> > > >> > > >> > > Check out the vibrant tech community on one of the
> world's
> > > most
> > > >> > > >> > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > > >> > > >> > > _______________________________________________
> > > >> > > >> > > Assp-test mailing list
> > > >> > > >> > > Assp-test@lists.sourceforge.net
> > > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> >
> ------------------------------------------------------------
> > > >> > > >> > ------------------
> > > >> > > >> > Check out the vibrant tech community on one of the world's
> > > most
> > > >> > > >> > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > > >> > > >> > _______________________________________________
> > > >> > > >> > Assp-test mailing list
> > > >> > > >> > Assp-test@lists.sourceforge.net
> > > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > DISCLAIMER:
> > > >> > > >> > *******************************************************
> > > >> > > >> > This email and any files transmitted with it may be
> > > confidential,
> > > >> > > >> legally
> > > >> > > >> > privileged and protected in law and are intended solely
> for
> > > the
> > > >> use
> > > >> > > of
> > > >> > > >> the
> > > >> > > >> >
> > > >> > > >> > individual to whom it is addressed.
> > > >> > > >> > This email was multiple times scanned for viruses. There
> > > should
> > > >> be
> > > >> > no
> > > >> > > >> > known virus in this email!
> > > >> > > >> > *******************************************************
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> ------------------------------------------------------------
> > > >> > > >> > ------------------
> > > >> > > >> > Check out the vibrant tech community on one of the world's
> > > most
> > > >> > > >> > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > > >> > > >> > _______________________________________________
> > > >> > > >> > Assp-test mailing list
> > > >> > > >> > Assp-test@lists.sourceforge.net
> > > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> ------------------------------------------------------------
> > > >> > > >> ------------------
> > > >> > > >> Check out the vibrant tech community on one of the world's
> > most
> > > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > > >> _______________________________________________
> > > >> > > >> Assp-test mailing list
> > > >> > > >> Assp-test@lists.sourceforge.net
> > > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> DISCLAIMER:
> > > >> > > >> *******************************************************
> > > >> > > >> This email and any files transmitted with it may be
> > > confidential,
> > > >> > > legally
> > > >> > > >> privileged and protected in law and are intended solely for
> > the
> > > use
> > > >> > of
> > > >> > > the
> > > >> > > >>
> > > >> > > >> individual to whom it is addressed.
> > > >> > > >> This email was multiple times scanned for viruses. There
> > should
> > > be
> > > >> no
> > > >> > > >> known virus in this email!
> > > >> > > >> *******************************************************
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> ------------------------------------------------------------
> > > >> > > >> ------------------
> > > >> > > >> Check out the vibrant tech community on one of the world's
> > most
> > > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > > >> _______________________________________________
> > > >> > > >> Assp-test mailing list
> > > >> > > >> Assp-test@lists.sourceforge.net
> > > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > > >>
> > > >> > > >>
> > > >> > > >
> > > >> > > ------------------------------------------------------------
> > > >> > > ------------------
> > > >> > > Check out the vibrant tech community on one of the world's most
> > > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > > _______________________________________________
> > > >> > > Assp-test mailing list
> > > >> > > Assp-test@lists.sourceforge.net
> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > DISCLAIMER:
> > > >> > > *******************************************************
> > > >> > > This email and any files transmitted with it may be
> confidential,
> > > >> > legally
> > > >> > > privileged and protected in law and are intended solely for the
> > use
> > > of
> > > >> > the
> > > >> > >
> > > >> > > individual to whom it is addressed.
> > > >> > > This email was multiple times scanned for viruses. There should
> > be
> > > no
> > > >> > > known virus in this email!
> > > >> > > *******************************************************
> > > >> > >
> > > >> > >
> > > >> > > ------------------------------------------------------------
> > > >> > > ------------------
> > > >> > > Check out the vibrant tech community on one of the world's most
> > > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > > _______________________________________________
> > > >> > > Assp-test mailing list
> > > >> > > Assp-test@lists.sourceforge.net
> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> > >
> > > >> > >
> > > >> > ------------------------------------------------------------
> > > >> > ------------------
> > > >> > Check out the vibrant tech community on one of the world's most
> > > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > _______________________________________________
> > > >> > Assp-test mailing list
> > > >> > Assp-test@lists.sourceforge.net
> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > DISCLAIMER:
> > > >> > *******************************************************
> > > >> > This email and any files transmitted with it may be confidential,
> > > >> legally
> > > >> > privileged and protected in law and are intended solely for the
> use
> > > of
> > > >> the
> > > >> >
> > > >> > individual to whom it is addressed.
> > > >> > This email was multiple times scanned for viruses. There should
> be
> > no
> > > >> > known virus in this email!
> > > >> > *******************************************************
> > > >> >
> > > >> >
> > > >> > ------------------------------------------------------------
> > > >> > ------------------
> > > >> > Check out the vibrant tech community on one of the world's most
> > > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> > _______________________________________________
> > > >> > Assp-test mailing list
> > > >> > Assp-test@lists.sourceforge.net
> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >> >
> > > >> >
> > > >> ------------------------------------------------------------
> > > >> ------------------
> > > >> Check out the vibrant tech community on one of the world's most
> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> _______________________________________________
> > > >> Assp-test mailing list
> > > >> Assp-test@lists.sourceforge.net
> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> DISCLAIMER:
> > > >> *******************************************************
> > > >> This email and any files transmitted with it may be confidential,
> > > legally
> > > >> privileged and protected in law and are intended solely for the use
> > of
> > > the
> > > >>
> > > >> individual to whom it is addressed.
> > > >> This email was multiple times scanned for viruses. There should be
> no
> > > >> known virus in this email!
> > > >> *******************************************************
> > > >>
> > > >>
> > > >> ------------------------------------------------------------
> > > >> ------------------
> > > >> Check out the vibrant tech community on one of the world's most
> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > >> _______________________________________________
> > > >> Assp-test mailing list
> > > >> Assp-test@lists.sourceforge.net
> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >>
> > > >>
> > > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > Assp-test@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > >
> > >
> > > DISCLAIMER:
> > > *******************************************************
> > > This email and any files transmitted with it may be confidential,
> > legally
> > > privileged and protected in law and are intended solely for the use of
> > the
> > >
> > > individual to whom it is addressed.
> > > This email was multiple times scanned for viruses. There should be no
> > > known virus in this email!
> > > *******************************************************
> > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > Assp-test@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
