these line are logged, if 'ConnectionLog' is enabled.
Thomas
Von: cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum: 06.10.2016 11:47
Betreff: Re: [Assp-test] unable to detect any running worker
WorkerLog is reduced now thanks.
Re the logging, something further in the file is changing this.
If I run grep against the file for something simple like the character 1
it
outputs hits all the way down to:
2016-10-06 06:19:17 [Worker_1] Worker_1 wakes up
It then stops and outputs "binary file matches" as it sees characters in
the stream that aren't consistent with the file type.
Are "bad line-end sequence" and "unsupported_SMTP_line_terminator" things
produced by the higher WorkerLog setting and will they go away now it is
set back to standard or do I have to change something else to suppress
them?
On Thu, Oct 6, 2016 at 9:50 AM, Thomas Eckardt
<thomas.ecka...@thockar.com>
wrote:
> >I'm wondering if the Ubuntu implementation just starts at the number 70
> for some reason.
>
> This is not possible. Global filehandle numbers are starting at 0, which
> is STDOUT on every OS.
> Don't care about as long as the process runs well.
>
> >I believe this is caused by ASSP outputting some raw SMTP data to the
> logfile.
>
> maillog.txt is an UTF-8 file (contains the UTF-8 BOM at its start).
>
> If there everyting is running well, reduce the WorkerLog setting.
>
> Thomas
>
>
>
> Von: cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum: 06.10.2016 10:38
> Betreff: Re: [Assp-test] unable to detect any running worker
>
>
>
> Hi Thomas,
>
> Thank you very much for your help and suggestions with this.
>
> So far both servers are running without shut down since I put 79_1 on
> yesterday.
>
> I am still seeing lots of can't interrupt worker - for example at
04:48:35
> when the server wasn't doing much I saw 70 lines logged within the same
> second. One server had a total of 3,500 logs between midnight and 9am.
>
> I'm also seeing issues with grep treating the maillog files as binary
> which
> I haven't seen before build 16270.
>
> I believe this is caused by ASSP outputting some raw SMTP data to the
> logfile. For example I see a load of none printable characters
immediately
> before the timestamp on a line that says 2016-10-06 07:01:16 [Worker_2]
> [unsupported_SMTP_line_terminator]
>
> Similarly I also see lines such as:
>
> 2016-10-06 07:01:29 [Worker_2] 5.10.252.206 info: bad line-end sequence
in
> ###
>
> Where ### is another load of none-printable characters.
>
> I presume this is someone running an exploit scanner as the IP address
> traces to Azerbaijan.
>
> All my logging is to standard at the moment except WorkerLog, is there
any
> way to have this information limited to the connection debug files so it
> doesn't make grep treat the maillog as a binary file?
>
> I had a think on the number of perl file handles. I can't explain it as
> all
> these servers do is run ASSP and exim4. I have a couple of nightly shell
> scripts but nothing that would run for more than a few seconds. ps only
> ever shows the ASSP perl process. I'm wondering if the Ubuntu
> implementation just starts at the number 70 for some reason.
>
> On Wed, Oct 5, 2016 at 5:52 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > 2.5.4 16279_1 is available at CVS /test.
> >
> > I tried to optimize the worker lookup process.
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von: cw <colin.war...@gmail.com>
> > An: ASSP development mailing list
<assp-test@lists.sourceforge.net>
> > Datum: 05.10.2016 18:08
> > Betreff: Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > So actually a development.
> >
> > It looks like disabling EnableHighPerformance has lightened the load a
> > little. I've gone an hour without a complete failure and it explains
why
> > I'm now seeing "Main_Thread can't interrupt Worker" instead of "unable
> to
> > detect any running worker".
> >
> > Presumably with high performance set to off ASSP has enough time to
> > recover
> > when the issue occurs. Hopefully this will buy me some leeway from
> having
> > to find an alternative and find the root cause of the issue.
> >
> > On Wed, Oct 5, 2016 at 4:40 PM, cw <colin.war...@gmail.com> wrote:
> >
> > > Unfortunately not. Most of the time the GUI is so unresponsive that
I
> > only
> > > go into it when necessary. I understand the performance impact on
it.
> > >
> > > Everything works well when ASSP starts, but within a few minutes it
> all
> > > starts to slow down and become unresponsive. I have a feeling that
it
> is
> > > either going to be something really simple or some odd combination
of
> > > events that I have been unfortunate enough to end up with.
> > >
> > > Most of the time I use command line and tail/grep/vi. Incidentally
> since
> > > yesterday I'm getting issues with grep reporting binary file matches
> on
> > the
> > > logs instead of outputting the results. I can only presume some of
the
> > > extra debug information is fooling grep but it doesn't help trying
to
> > track
> > > down the right lights in the log!
> > >
> > > On Wed, Oct 5, 2016 at 4:32 PM, Thomas Eckardt
> > <thomas.ecka...@thockar.com
> > > > wrote:
> > >
> > >> >2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > interrupted
> > >> >Worker_4 in 10.758 seconds - got (ok)
> > >>
> > >> I'm looking for the reason of this very huge time value of 10.758
> > seconds.
> > >>
> > >> The only reason I can find is - you are running constantly at least
> one
> > >> auto refreshing GUI screen. I'm right?
> > >>
> > >> switch to 'tail' the maillog.txt on a console session
> > >>
> > >> tail -fn 30 /usr...../maillog.txt
> > >>
> > >> do not use the shutdown_list (connection screen)!
> > >>
> > >> How it works now?
> > >>
> > >> Thomas
> > >>
> > >>
> > >>
> > >> Von: cw <colin.war...@gmail.com>
> > >> An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> > >> Datum: 05.10.2016 17:05
> > >> Betreff: Re: [Assp-test] unable to detect any running worker
> > >>
> > >>
> > >>
> > >> I specifically checked netstat for established connections and
didn't
> > >> include any in the wait state or part open state. According to the
OS
> > >> these
> > >> are connections still fully open. I excluded the ones to the MTA so
> > this
> > >> should be purely inbound SMTP connections.
> > >>
> > >> I've set EnableHighPerformance back to off.
> > >>
> > >> Loads of runs produced:
> > >>
> > >> 0.003068
> > >> 0.003083
> > >> 0.003115
> > >> 0.003108
> > >> 0.003101
> > >> 0.003095
> > >> 0.0031
> > >> 0.003096
> > >> 0.003096
> > >> 0.003096
> > >> 0.003105
> > >> 0.003099
> > >> 0.003097
> > >> 0.003095
> > >> 0.00308
> > >> 0.003071
> > >> 0.003071
> > >> 0.003083
> > >> 0.003077
> > >> 0.003072
> > >> 0.003076
> > >> 0.003075
> > >> 0.003075
> > >> 0.003072
> > >> 0.0031
> > >> 0.003077
> > >> 0.003094
> > >> 0.003099
> > >>
> > >>
> > >> On Wed, Oct 5, 2016 at 3:45 PM, Thomas Eckardt
> > >> <thomas.ecka...@thockar.com>
> > >> wrote:
> > >>
> > >> > >I'm currently comparing things. shutdown_list shows 5 smtp
> sessions
> > in
> > >> > >threads whilst netstat shows 18 connections established to port
25
> > from
> > >> > the
> > >> > >outside world.
> > >> > >It then went down to 2 SMTP sessions in threads and netstat
showed
> > 13
> > >> > >connections.
> > >> >
> > >> > the state of most of the connection should than be something like
> > >> > "fin_wait". Closed connection are keeped by the OS socket handler
> for
> > >> two
> > >> > minutes to satisfy the peer, if there are still pakets on the
way.
> > >> >
> > >> > set 'EnableHighPerformance' back to the default - 'OFF'
> > >> >
> > >> >
> > >> > please run the following small perl script and tell me the
output,
> > >> > Possibly run it multiple times. Is the sleep time very
different?
> > >> > I expect to see a sleep time of 0.003 to 0.005 seconds
> > >> >
> > >> > use strict;
> > >> > use Time::HiRes;
> > >> >
> > >> > my $t = 3000/1000000;
> > >> > print Time::HiRes::sleep($t);
> > >> >
> > >> > Thomas
> > >> >
> > >> >
> > >> > Von: cw <colin.war...@gmail.com>
> > >> > An: ASSP development mailing list
> > <assp-test@lists.sourceforge.net>
> > >> > Datum: 05.10.2016 16:27
> > >> > Betreff: Re: [Assp-test] unable to detect any running
worker
> > >> >
> > >> >
> > >> >
> > >> > ThreadCycleTime is set to 3000 microseconds
> > >> > EnableHighPerformance I have changed during the course of this
week
> > with
> > >> > no
> > >> > effect on the problem. It was at default and is now at the
highest
> > >> > setting.
> > >> > I'd check the exact names for the settings except neither server
> will
> > >> give
> > >> > me a connection to the GUI at the moment.
> > >> >
> > >> > I'm currently comparing things. shutdown_list shows 5 smtp
sessions
> > in
> > >> > threads whilst netstat shows 18 connections established to port
25
> > from
> > >> > the
> > >> > outside world.
> > >> > It then went down to 2 SMTP sessions in threads and netstat
showed
> 13
> > >> > connections.
> > >> >
> > >> > On Wed, Oct 5, 2016 at 3:06 PM, Thomas Eckardt
> > >> > <thomas.ecka...@thockar.com>
> > >> > wrote:
> > >> >
> > >> > > What are your setting for :
> > >> > >
> > >> > > ThreadCycleTime
> > >> > > EnableHighPerformance
> > >> > >
> > >> > > Thomas
> > >> > >
> > >> > >
> > >> > >
> > >> > > Von: cw <colin.war...@gmail.com>
> > >> > > An: ASSP development mailing list
> > >> <assp-test@lists.sourceforge.net>
> > >> > > Datum: 05.10.2016 15:51
> > >> > > Betreff: Re: [Assp-test] unable to detect any running
> worker
> > >> > >
> > >> > >
> > >> > >
> > >> > > I can confirm that this number is artificially high quite
easily.
> > >> > > I actually have a third ASSP instance. This one is specially
> > >> configured
> > >> > to
> > >> > > only accept emails from Office 365 and acts as an outbound
relay
> > >> purely
> > >> > to
> > >> > > gather messages for the corpus. This one does not experience
the
> > >> problem
> > >> > > with no running workers being available.
> > >> > >
> > >> > > Its stats haven't been reset for a long time and show an
average
> of
> > >> 2901
> > >> > > per day. Concurrent SMTP sessions since last restart are 0 (4
> max).
> > >> > >
> > >> > > The last message showing in the logs says:
> > >> > >
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread got
> connection
> > >> > request
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread looks up
the
> > best
> > >> > > Worker for new connection - 75
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread will wait
> (max
> > 30
> > >> s)
> > >> > > for the answer of Worker_7 which handles 0 sockets
> > >> > > 2016-10-05 14:43:57 [Worker_7] Worker_7 wakes up
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread freed by
idle
> > >> > Worker_7
> > >> > > in 0.005 seconds - got (ok)
> > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 got connection
from
> > >> > > MainThread - 75/75
> > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 freed Main_Thread
-
> > 84
> > >> > >
> > >> > > There are no other emails going on at this time. Just the one
> > >> connection
> > >> > > and it still says 75. This server is firewalled and the only
> > messages
> > >> it
> > >> > > ever sees are from known Office 365 IP addresses along with the
> > >> internal
> > >> > > test server that sends 1 message every 60 seconds.
> > >> > >
> > >> > > On Wed, Oct 5, 2016 at 2:33 PM, cw <colin.war...@gmail.com>
> wrote:
> > >> > >
> > >> > > > Hi Thomas,
> > >> > > >
> > >> > > > Thank you for the rundown. The load is exceptionally high at
> the
> > >> > moment
> > >> > > > because everything backs up every time the mail servers stop.
> > Having
> > >> > > said
> > >> > > > that, I've just watched it happen again on one of the
servers.
> It
> > >> only
> > >> > > had
> > >> > > > 7 messages showing in the shutdown_list when it happened.
> Looking
> > at
> > >> > the
> > >> > > > logs I see:
> > >> > > >
> > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection
> from
> > >> > > > MainThread - 78/78
> > >> > > > 2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > >> > interrupted
> > >> > > > Worker_4 in 10.758 seconds - got (ok)
> > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed
Main_Thread
> -
> > >> 101
> > >> > > >
> > >> > > > So something is artificially inflating the connection count.
My
> > best
> > >> > > guess
> > >> > > > ties in with something else I was looking at where a remote
> > server
> > >> was
> > >> > > > opening two connections that never got answered. It only got
> > through
> > >> > on
> > >> > > the
> > >> > > > third connection and that could be replicated each time. I
put
> it
> > >> down
> > >> > > to
> > >> > > > some odd firewall on their end but maybe I was wrong in doing
> > that.
> > >> > > >
> > >> > > > The actual stats on one mailserver are:
> > >> > > >
> > >> > > > Last reset 8 days, 0 hours, 6 mins
> > >> > > > Messages process 165413 (20664 per day)
> > >> > > > SMTP Connections received 113278
> > >> > > >
> > >> > > > The other is:
> > >> > > > Last reset 8 days, 0 hours, 18 mins
> > >> > > > Messaged processed 131720 (16438 per day)
> > >> > > > SMTP Connections received 93088
> > >> > > >
> > >> > > > The stats reset themselves during on of the upgrades of ASSP,
> not
> > >> sure
> > >> > > why.
> > >> > > >
> > >> > > > Each server is 16 CPU cores, 12GB RAM. DB is on a separate
> MySQL
> > box
> > >> > > > though MTA and Clam run local. URIBL is disabled. I don't use
> > OCR,
> > >> DCC
> > >> > > or
> > >> > > > Razor. I tried to get OCR working once and all hell broke
lose
> > and I
> > >> > > > haven't been back since.
> > >> > > >
> > >> > > > I've been monitoring the performance of the servers all along
> and
> > >> not
> > >> > > > seeing any bottlenecks in CPU, RAM or network.When the
problem
> > >> started
> > >> > > the
> > >> > > > mail servers had 8 and 10GB RAM respectively (10GB for the
one
> > >> running
> > >> > > > rebuild).
> > >> > > > All the best,
> > >> > > > Colin
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt
> > >> > > <thomas.ecka...@thockar.com
> > >> > > > > wrote:
> > >> > > >
> > >> > > >> It seems, that this system is overloaded with active
> > connections.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed
by
> > >> > > interrupted
> > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > >> > > >>
> > >> > > >> Even this time value is ten times higher than expected.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> Main_Thread
> > -
> > >> > 170
> > >> > > >>
> > >> > > >> The '170' is the filehandle number in perl (counted from
> 1.....)
> > -
> > >> > this
> > >> > > >> implies 70 or more active connections.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will
wait
> > (max
> > >> > 30
> > >> > > s)
> > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > >> > > >>
> > >> > > >> This implies the same, if the equal balancing of all
> connections
> > >> over
> > >> > > all
> > >> > > >> seven workers works correctly.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2
seconds
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.103252172470093
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > >> 0.0278699398040771
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > >> 0.0423040390014648
> > >> > > >>
> > >> > > >> The processing time itself seems to be normal. SC-Time - is
> the
> > >> time
> > >> > > >> required to process one loop (read + queue or read + check +
> > queue)
> > >> > for
> > >> > > >> one active connection.
> > >> > > >>
> > >> > > >> Assuming an average of 4 second life time per connection -
the
> > >> > workload
> > >> > > >> per day would be 1.5 million (70 / 4 *3600 *24) connections
> > based
> > >> on
> > >> > > the
> > >> > > >> provided output.
> > >> > > >> Configured to an ISP mode the avg. maximum for a single assp
> > >> instance
> > >> > > is
> > >> > > >> 800.000 connections. To be able to handle workload peeks,
the
> > >> > > >> configuration should not allow more than 400.000 connection
a
> > day.
> > >> > > >>
> > >> > > >> ISP mode:
> > >> > > >> - 16 CPU cores
> > >> > > >> - 16 Workers
> > >> > > >> - 24 GB RAM
> > >> > > >> - dedicated systems for MTA, enterprise database, ClamAV
> > >> > > >> - HMM and spamDB holded unshared in RAM
> > >> > > >> - high performance DNS servers
> > >> > > >> - limited DNS usage (disable most extensive DNS-using checks
-
> > eg.
> > >> > > URIBL)
> > >> > > >> - no OCR
> > >> > > >> - no DCC
> > >> > > >> - no Razor2
> > >> > > >>
> > >> > > >> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed
by
> > >> > > interrupted
> > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > >> > > >>
> > >> > > >> This time value is near the end of the line. At this time a
> > small
> > >> > count
> > >> > > of
> > >> > > >> additionally connections can lead in to an assp shutdown,
> > because
> > >> the
> > >> > > >> workers don't accept new connections.
> > >> > > >>
> > >> > > >> Thomas
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> Von: cw <colin.war...@gmail.com>
> > >> > > >> An: ASSP development mailing list
> > >> > <assp-test@lists.sourceforge.net>
> > >> > > >> Datum: 05.10.2016 13:15
> > >> > > >> Betreff: Re: [Assp-test] unable to detect any running
> > worker
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> Thanks.
> > >> > > >>
> > >> > > >> I've had both servers come up against unable to detect any
> > running
> > >> > > worker
> > >> > > >> since clearing out all the files suggested. So I'm getting
> 16279
> > >> > > running
> > >> > > >> now.
> > >> > > >>
> > >> > > >> I noticed the startup with those files removed was really
> quick,
> > >> > > starting
> > >> > > >> back up the second time took several minutes so presumably
> > reading
> > >> > > those
> > >> > > >> files during startup takes a little while versus creating
them
> > >> fresh.
> > >> > > >>
> > >> > > >> I've caught one already and traced it through the new logs:
> > >> > > >>
> > >> > > >> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got
> > connection
> > >> > > request
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up
> the
> > >> best
> > >> > > >> Worker for new connection - 73
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt
> worker
> > >> > > Worker_3
> > >> > > >> (12) for new connection
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread
> interrupted
> > >> > > Worker_3
> > >> > > >> (12) to submit the connection
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will
wait
> > (max
> > >> 30
> > >> > > s)
> > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3:
> > 0.0462169647216797
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection
> > from
> > >> > > >> MainThread - 73/73
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> Main_Thread
> > -
> > >> 170
> > >> > > >> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Connected:
session:7FF3AA6E9A28
> > >> > > >> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.103252172470093
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.0278699398040771
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply]
220
> > >> > > >> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct
2016
> > >> > 11:47:41
> > >> > > >> +0100
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.0423040390014648
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time
Worker_3:
> > >> > > >> 0.0296478271484375
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2
seconds
> > >> > > >> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > >> > > >>
> > >> > > >> In this case, it looks like the connection ended without any
> > actual
> > >> > > data.
> > >> > > >> There is lots of activity reported by Worker_3 in between
> > 11:47:43
> > >> > and
> > >> > > >> 11:48:13 but this all pertains to other connections that
were
> > >> already
> > >> > > in
> > >> > > >> progress.
> > >> > > >>
> > >> > > >> That is a bit different to the earlier one where a message
was
> > >> > received
> > >> > > >> and
> > >> > > >> the message completed within the 30s window.
> > >> > > >>
> > >> > > >> I'm not seeing anything to help me figure out why though and
I
> > >> don't
> > >> > > want
> > >> > > >> to simply post a big excerpt of the maillog.txt.
> > >> > > >>
> > >> > > >> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
> > >> > > >> <thomas.ecka...@thockar.com>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > >I looked at SF but only see 16275 in test (updated 3 days
> > ago)
> > >> so
> > >> > > maybe
> > >> > > >> > it
> > >> > > >> > hasn't made its way live yet.
> > >> > > >> >
> > >> > > >> > Sorry, my background CVS sync was not running - update is
> > done.
> > >> > > >> >
> > >> > > >> > Thomas
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > Von: cw <colin.war...@gmail.com>
> > >> > > >> > An: ASSP development mailing list
> > >> > > <assp-test@lists.sourceforge.net>
> > >> > > >> > Datum: 05.10.2016 11:48
> > >> > > >> > Betreff: Re: [Assp-test] unable to detect any
running
> > >> worker
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > Thank you Thomas.
> > >> > > >> >
> > >> > > >> > useDB4IntCache - already set to off
> > >> > > >> > I've set WorkerLog to diagnostic and done the other steps.
> > >> > > >> > I don't have anything in CorrectASSPcfg.pm so as part of
the
> > >> > > >> > troubleshooting I have previously deleted it and
downloaded
> a
> > >> fresh
> > >> > > copy
> > >> > > >> > from SourceForge.
> > >> > > >> >
> > >> > > >> > I have to say that ASSP started up within seconds after
> > clearing
> > >> > > those
> > >> > > >> > files out - it normally takes several minutes and has
always
> > >> done.
> > >> > > >> >
> > >> > > >> > I looked at SF but only see 16275 in test (updated 3 days
> ago)
> > so
> > >> > > maybe
> > >> > > >> it
> > >> > > >> > hasn't made its way live yet.
> > >> > > >> >
> > >> > > >> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> > >> > > >> > <thomas.ecka...@thockar.com>
> > >> > > >> > wrote:
> > >> > > >> >
> > >> > > >> > > I've provided an updated assp.pl (2.5.4 16279) in CVS
> /test.
> > >> This
> > >> > > >> > version
> > >> > > >> > > shows some more information, if 'WorkerLog' is set to
> > >> diagnostic.
> > >> > > >> > >
> > >> > > >> > > Thomas
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > Von: cw <colin.war...@gmail.com>
> > >> > > >> > > An: ASSP development mailing list
> > >> > > >> <assp-test@lists.sourceforge.net>
> > >> > > >> > > Datum: 05.10.2016 10:10
> > >> > > >> > > Betreff: Re: [Assp-test] unable to detect any
> running
> > >> > worker
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > Hi Thomas,
> > >> > > >> > >
> > >> > > >> > > Thanks for chipping in. All modules are installed by
> running
> > >> the
> > >> > > >> latest
> > >> > > >> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in
> it
> > so
> > >> it
> > >> > > >> > requires
> > >> > > >> > > a minor edit to the code to get it to build. So that
> module
> > was
> > >> > > >> > installed
> > >> > > >> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and
> > >> running:
> > >> > > >> > > make clean
> > >> > > >> > > perl Makefile.PL
> > >> > > >> > > make
> > >> > > >> > > make test
> > >> > > >> > > make install
> > >> > > >> > >
> > >> > > >> > > I run cpan-outdated -p|cpanm from time to time in order
to
> > keep
> > >> > > >> modules
> > >> > > >> > up
> > >> > > >> > > to date as well so things don't stay stuck on old
> versions.
> > >> > > >> > >
> > >> > > >> > > Everything that is installed now is a completely fresh
> build
> > as
> > >> > the
> > >> > > >> > > Upgrade
> > >> > > >> > > to 16.04 replaced perl 5.18 with 5.22 and all the
modules
> > >> > therefore
> > >> > > >> had
> > >> > > >> > to
> > >> > > >> > > be installed from scratch.
> > >> > > >> > >
> > >> > > >> > > I bypassed the issue by truncating the tables and
setting
> up
> > >> the
> > >> > > users
> > >> > > >> > > again. With there only being one user it was easier to
do
> > that
> > >> > than
> > >> > > >> muck
> > >> > > >> > > about with it - especially with the other bigger issue
at
> > hand.
> > >> > > >> > >
> > >> > > >> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > >> > > >> > > <thomas.ecka...@thockar.com>
> > >> > > >> > > wrote:
> > >> > > >> > >
> > >> > > >> > > > >to replace the corrupted encrypted
> > >> > > >> > > > strings with the correct values
> > >> > > >> > > >
> > >> > > >> > > > Was the 'Crypt::GOST' module from the SF download page
> at
> > the
> > >> > old
> > >> > > >> assp
> > >> > > >> > > > instance?
> > >> > > >> > > > If it was, did you install the 'Crypt::GOST' module
from
> > the
> > >> SF
> > >> > > >> > download
> > >> > > >> > > > page, before you started the new assp instance?
> > >> > > >> > > >
> > >> > > >> > > > https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > >> > > >> > > >
> > >> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > >> > > >> > > >
> > >> > > >> > > > Thomas
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > Von: cw <colin.war...@gmail.com>
> > >> > > >> > > > An: ASSP development mailing list
> > >> > > >> > <assp-test@lists.sourceforge.net>
> > >> > > >> > > > Datum: 05.10.2016 09:52
> > >> > > >> > > > Betreff: Re: [Assp-test] unable to detect any
> > running
> > >> > > worker
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > Cheers for the reply.
> > >> > > >> > > >
> > >> > > >> > > > I don't think backups are an option seen as I've moved
> > >> > completely
> > >> > > >> from
> > >> > > >> > > > Ubuntu 14.04 to Ubuntu 16.04.
> > >> > > >> > > >
> > >> > > >> > > > Also, this issue has been around for months. It was
> > causing a
> > >> > > >> handful
> > >> > > >> > of
> > >> > > >> > > > shutdowns a week with the occasional spat of more
> frequent
> > >> > > >> shutdowns.
> > >> > > >> > It
> > >> > > >> > > > is
> > >> > > >> > > > entirely possible that the errors are behaviour
related
> > and
> > >> > > nothing
> > >> > > >> to
> > >> > > >> > > do
> > >> > > >> > > > with the upgrade and that current email behaviour is
> > >> triggering
> > >> > > it
> > >> > > >> big
> > >> > > >> > > > style.
> > >> > > >> > > >
> > >> > > >> > > > I'm not convinced though, unfortunately I'm not
> convinced
> > of
> > >> > > >> anything
> > >> > > >> > > else
> > >> > > >> > > > hence not having much to go on. I can't see any
> > consistencies
> > >> > in
> > >> > > the
> > >> > > >> > > > behaviour leading up to the events.
> > >> > > >> > > >
> > >> > > >> > > > I don't think it is database related. It happened on
one
> > of
> > >> the
> > >> > > mail
> > >> > > >> > > > servers during the upgrade to 16.04 when ASSP had just
> > >> started
> > >> > > but I
> > >> > > >> > had
> > >> > > >> > > > not yet got into the web interface to replace the
> > corrupted
> > >> > > >> encrypted
> > >> > > >> > > > strings with the correct values so all database
> > connections
> > >> > were
> > >> > > in
> > >> > > >> > > error.
> > >> > > >> > > >
> > >> > > >> > > > The problem has already started again this morning so
I
> > can
> > >> see
> > >> > > this
> > >> > > >> > > being
> > >> > > >> > > > another fun day that either leads to a fix or having
to
> > put
> > >> > > >> something
> > >> > > >> > > else
> > >> > > >> > > > in place.
> > >> > > >> > > >
> > >> > > >> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post
> > <nntp.p...@gmail.com>
> > >> > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > > I've been reading here, but I haven't had anything
to
> > >> > suggest.
> > >> > > All
> > >> > > >> > > > seems
> > >> > > >> > > > > quite odd if it was working prior to upgrading and
> > >> > downgrading
> > >> > > >> > didn't
> > >> > > >> > > > work.
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > Could you spin up a backup of the installation after
> > >> copying
> > >> > > the
> > >> > > >> > > current
> > >> > > >> > > > > data? Sure you'd have an older corpus, but I'd
think
> > you
> > >> > could
> > >> > > >> add
> > >> > > >> > > the
> > >> > > >> > > > new
> > >> > > >> > > > > files if necessary, manually replace whitelist etc.
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw
> > <colin.war...@gmail.com
> > >> >
> > >> > > >> wrote:
> > >> > > >> > > > >
> > >> > > >> > > > > > Further development on this today, very little.
> > >> > > >> > > > > > I have moved both servers onto Ubuntu 16.04 LTS
> which
> > >> means
> > >> > > >> going
> > >> > > >> > > from
> > >> > > >> > > > > perl
> > >> > > >> > > > > > 5.18 to 5.22 and rebuilding all perl modules from
> > >> scratch.
> > >> > > >> > > > > >
> > >> > > >> > > > > > The admin user db did not work after the upgrade
so
> I
> > had
> > >> > to
> > >> > > >> empty
> > >> > > >> > > the
> > >> > > >> > > > > > tables before it would come back online.
> > >> > > >> > > > > >
> > >> > > >> > > > > > I'm still getting delayed emails and assp shutting
> > down
> > >> > > telling
> > >> > > >> me
> > >> > > >> > > it
> > >> > > >> > > > is
> > >> > > >> > > > > > unable
> > >> > > >> > > > > > to detect any running worker.
> > >> > > >> > > > > >
> > >> > > >> > > > > > If this goes on much longer the MD will pull the
> plug
> > and
> > >> > > we'll
> > >> > > >> > end
> > >> > > >> > > up
> > >> > > >> > > > > > moving to a third party solution which is not
> > something I
> > >> > > want
> > >> > > >> but
> > >> > > >> > > if
> > >> > > >> > > > I
> > >> > > >> > > > > > can't fix it I can't defend it :/
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > ------------------------------------------------------------
> > >> > > >> > > > > > ------------------
> > >> > > >> > > > > > Check out the vibrant tech community on one of the
> > >> world's
> > >> > > most
> > >> > > >> > > > > > engaging tech sites, SlashDot.org!
> > >> http://sdm.link/slashdot
> > >> > > >> > > > > > _______________________________________________
> > >> > > >> > > > > > Assp-test mailing list
> > >> > > >> > > > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > > >
> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> ------------------------------------------------------------
> > >> > > >> > > > > ------------------
> > >> > > >> > > > > Check out the vibrant tech community on one of the
> > world's
> > >> > most
> > >> > > >> > > > > engaging tech sites, SlashDot.org!
> > >> http://sdm.link/slashdot
> > >> > > >> > > > > _______________________________________________
> > >> > > >> > > > > Assp-test mailing list
> > >> > > >> > > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > >
https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > >
> > ------------------------------------------------------------
> > >> > > >> > > > ------------------
> > >> > > >> > > > Check out the vibrant tech community on one of the
> world's
> > >> most
> > >> > > >> > > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > >> > > >> > > > _______________________________________________
> > >> > > >> > > > Assp-test mailing list
> > >> > > >> > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > DISCLAIMER:
> > >> > > >> > > >
*******************************************************
> > >> > > >> > > > This email and any files transmitted with it may be
> > >> > confidential,
> > >> > > >> > > legally
> > >> > > >> > > > privileged and protected in law and are intended
solely
> > for
> > >> the
> > >> > > use
> > >> > > >> of
> > >> > > >> > > the
> > >> > > >> > > >
> > >> > > >> > > > individual to whom it is addressed.
> > >> > > >> > > > This email was multiple times scanned for viruses.
There
> > >> should
> > >> > > be
> > >> > > >> no
> > >> > > >> > > > known virus in this email!
> > >> > > >> > > >
*******************************************************
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > ------------------------------------------------------------
> > >> > > >> > > > ------------------
> > >> > > >> > > > Check out the vibrant tech community on one of the
> world's
> > >> most
> > >> > > >> > > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > >> > > >> > > > _______________________________________________
> > >> > > >> > > > Assp-test mailing list
> > >> > > >> > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> ------------------------------------------------------------
> > >> > > >> > > ------------------
> > >> > > >> > > Check out the vibrant tech community on one of the
world's
> > most
> > >> > > >> > > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > >> > > >> > > _______________________________________________
> > >> > > >> > > Assp-test mailing list
> > >> > > >> > > Assp-test@lists.sourceforge.net
> > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > DISCLAIMER:
> > >> > > >> > > *******************************************************
> > >> > > >> > > This email and any files transmitted with it may be
> > >> confidential,
> > >> > > >> > legally
> > >> > > >> > > privileged and protected in law and are intended solely
> for
> > the
> > >> > use
> > >> > > of
> > >> > > >> > the
> > >> > > >> > >
> > >> > > >> > > individual to whom it is addressed.
> > >> > > >> > > This email was multiple times scanned for viruses. There
> > should
> > >> > be
> > >> > > no
> > >> > > >> > > known virus in this email!
> > >> > > >> > > *******************************************************
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> ------------------------------------------------------------
> > >> > > >> > > ------------------
> > >> > > >> > > Check out the vibrant tech community on one of the
world's
> > most
> > >> > > >> > > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > >> > > >> > > _______________________________________________
> > >> > > >> > > Assp-test mailing list
> > >> > > >> > > Assp-test@lists.sourceforge.net
> > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> >
------------------------------------------------------------
> > >> > > >> > ------------------
> > >> > > >> > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > engaging tech sites, SlashDot.org!
http://sdm.link/slashdot
> > >> > > >> > _______________________________________________
> > >> > > >> > Assp-test mailing list
> > >> > > >> > Assp-test@lists.sourceforge.net
> > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > DISCLAIMER:
> > >> > > >> > *******************************************************
> > >> > > >> > This email and any files transmitted with it may be
> > confidential,
> > >> > > >> legally
> > >> > > >> > privileged and protected in law and are intended solely
for
> > the
> > >> use
> > >> > > of
> > >> > > >> the
> > >> > > >> >
> > >> > > >> > individual to whom it is addressed.
> > >> > > >> > This email was multiple times scanned for viruses. There
> > should
> > >> be
> > >> > no
> > >> > > >> > known virus in this email!
> > >> > > >> > *******************************************************
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
------------------------------------------------------------
> > >> > > >> > ------------------
> > >> > > >> > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > engaging tech sites, SlashDot.org!
http://sdm.link/slashdot
> > >> > > >> > _______________________________________________
> > >> > > >> > Assp-test mailing list
> > >> > > >> > Assp-test@lists.sourceforge.net
> > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> >
> > >> > > >> >
> > >> > > >> ------------------------------------------------------------
> > >> > > >> ------------------
> > >> > > >> Check out the vibrant tech community on one of the world's
> most
> > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> _______________________________________________
> > >> > > >> Assp-test mailing list
> > >> > > >> Assp-test@lists.sourceforge.net
> > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> DISCLAIMER:
> > >> > > >> *******************************************************
> > >> > > >> This email and any files transmitted with it may be
> > confidential,
> > >> > > legally
> > >> > > >> privileged and protected in law and are intended solely for
> the
> > use
> > >> > of
> > >> > > the
> > >> > > >>
> > >> > > >> individual to whom it is addressed.
> > >> > > >> This email was multiple times scanned for viruses. There
> should
> > be
> > >> no
> > >> > > >> known virus in this email!
> > >> > > >> *******************************************************
> > >> > > >>
> > >> > > >>
> > >> > > >> ------------------------------------------------------------
> > >> > > >> ------------------
> > >> > > >> Check out the vibrant tech community on one of the world's
> most
> > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> _______________________________________________
> > >> > > >> Assp-test mailing list
> > >> > > >> Assp-test@lists.sourceforge.net
> > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >>
> > >> > > >>
> > >> > > >
> > >> > > ------------------------------------------------------------
> > >> > > ------------------
> > >> > > Check out the vibrant tech community on one of the world's most
> > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > _______________________________________________
> > >> > > Assp-test mailing list
> > >> > > Assp-test@lists.sourceforge.net
> > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > DISCLAIMER:
> > >> > > *******************************************************
> > >> > > This email and any files transmitted with it may be
confidential,
> > >> > legally
> > >> > > privileged and protected in law and are intended solely for the
> use
> > of
> > >> > the
> > >> > >
> > >> > > individual to whom it is addressed.
> > >> > > This email was multiple times scanned for viruses. There should
> be
> > no
> > >> > > known virus in this email!
> > >> > > *******************************************************
> > >> > >
> > >> > >
> > >> > > ------------------------------------------------------------
> > >> > > ------------------
> > >> > > Check out the vibrant tech community on one of the world's most
> > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > _______________________________________________
> > >> > > Assp-test mailing list
> > >> > > Assp-test@lists.sourceforge.net
> > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > >
> > >> > >
> > >> > ------------------------------------------------------------
> > >> > ------------------
> > >> > Check out the vibrant tech community on one of the world's most
> > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > _______________________________________________
> > >> > Assp-test mailing list
> > >> > Assp-test@lists.sourceforge.net
> > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > DISCLAIMER:
> > >> > *******************************************************
> > >> > This email and any files transmitted with it may be confidential,
> > >> legally
> > >> > privileged and protected in law and are intended solely for the
use
> > of
> > >> the
> > >> >
> > >> > individual to whom it is addressed.
> > >> > This email was multiple times scanned for viruses. There should
be
> no
> > >> > known virus in this email!
> > >> > *******************************************************
> > >> >
> > >> >
> > >> > ------------------------------------------------------------
> > >> > ------------------
> > >> > Check out the vibrant tech community on one of the world's most
> > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > _______________________________________________
> > >> > Assp-test mailing list
> > >> > Assp-test@lists.sourceforge.net
> > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> >
> > >> >
> > >> ------------------------------------------------------------
> > >> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> Assp-test mailing list
> > >> Assp-test@lists.sourceforge.net
> > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >>
> > >>
> > >>
> > >>
> > >> DISCLAIMER:
> > >> *******************************************************
> > >> This email and any files transmitted with it may be confidential,
> > legally
> > >> privileged and protected in law and are intended solely for the use
> of
> > the
> > >>
> > >> individual to whom it is addressed.
> > >> This email was multiple times scanned for viruses. There should be
no
> > >> known virus in this email!
> > >> *******************************************************
> > >>
> > >>
> > >> ------------------------------------------------------------
> > >> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> Assp-test mailing list
> > >> Assp-test@lists.sourceforge.net
> > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >>
> > >>
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
