>ASSP will accept connections and hold them for anywhere from a few seconds up to 10-20 seconds and then carry on
This is by design. Let's have a look at one single SMTP worker. Assume it has 3 active connections at a point in time and is just doing something (eg a check) for one of the connections. Now the MainThread gets a new connection. It tries to find a SMTP worker, which could get this connection (anyway: per interrupt or regular) The worker we looking at, gets interrupted in its operation or freeup the MainThread regular and graps the connection (here you see the blank screen in telnet) The worker has now 3 active connections and one new - and returns back to the operation where it was interrupted The SMTP worker finishes one operation for all his known three active connections (one cycle - read data->process, write data). Now it has 4 active connections and processes them in random order - each one , one cycle. The first action for the new graped connection is to read the greeting from the MTA and to send it to the client - you'll see the '220 welcome.....' in the telnet session. So, the time between the blank screen (connect) and the greeting (first send) depends on the work that is to do for the still existing connections. This could be some milliseconds or even several seconds. There is nothing you have to worry about - the maximum time between two read/write operations is defined by the SMTP-timeout value (assp or MTA) - typical 180 seconds, most times more. So any delay between 0 and 180 seconds could be happen and is (and has to be !!!) acceptable for all systems that are doing SMTP. It is conceivalbe, that a connection is running in to a SMTP timeout because of a temporary very high assp workload - and in this case an assp restart does not really help nor is this a good reason to do it. The maximum SMTP timeout used by assp for some connection types is 1200 seconds ! I've spotted this several time - and I say it again! ***************************************************** !!!! NOTICE NOTICE NOTICE !!!!! For this reason, it is NOT recommended (IMHO it is NONSENSE) to monitor assp at any SMTP listener. The statistic port (55553) could be used, SNMP could be used, syslog could be used (assp-monitor.pl uses syslog for example). NEVER EVER use the web listerner 55555 to monitor assp - this can lead in to unexpected config changes or config reloads - in worth case you can lose parts or the complete configuration. It is also a bad idea to 'tail' the maillog.txt for monitoring, because it is possible that an idle assp instance does not write anything in to the maillog.txt for several minutes. !!!! NOTICE NOTICE NOTICE !!!!! ***************************************************** I monitor my ASSP instances with assp-monitor.pl since several years. >For example I intended to build a MySQL cluster What we saw in the last days on a high workload assp cluster (15 assp each with 15 workers): An MySQL instance with 32 CPU's and several hundred GB of RAM was unable to handle the assp DB workload!!!! This caused the MySQL instance to max out the CPU's. I don't think that any affordable database engine is able to do the workload. Because assp produces hundreds of different sequentel SQL queries, no record or query result caching will help. The only thing that would help is to hold the complete database data in memory (RAM). But in this case, the buildin hash mechanims of Perl is very much faster than any DB engine - and we already have it active - we only need to use it. How ever, we have to share the data between multiple instances. The concept for the ISP mode is to hold the very large databases (HMMdb and Spamdb) in Perl's internal very fast hash mechanism (native in each worker) in RAM and to share them file based. The rebuildspamdb instance distributes the resulting files to all other instances, which will then refresh/reload there internal databases. All the other data, like pb, whitelist and so on, are shared in a RDB (MySQL) instance. The averaged workload for these tables is manageable and could be done at relative low cost. Now let's do some smart calculation. Assume we have 17 SMTP workers - this results in 20 threads in sum. HMM and Bayesian have 3 million records in sum, which consumes an avg. of 400 MB RAM in each thread -> 400MB * 20 = 8GB RAM only for these two data tables. If the rebuild is finished, assp has to replace all the 8GB RAM without any disturbance (out of memory, SMTP timeout, memory leaking, being unresponsive .....). This is the real challenge! But for now - this is working well (not perfect !). Thomas Von: Colin Waring <co...@dolphinict.co.uk> An: ASSP development mailing list <assp-test@lists.sourceforge.net> Datum: 09.03.2015 10:08 Betreff: Re: [Assp-test] fixes in assp 2.4.4 build 15067 Thanks for the explanation Thomas, Most of the changes I've been making are aimed at redundancy over performance. For example I intended to build a MySQL cluster and put it behind a load balancer so that we can handle the DB server going offline for maintenance etc. I do have one issue that I've never been sure about whether it is performance related. Quite regularly, ASSP will accept connections and hold them for anywhere from a few seconds up to 10-20 seconds and then carry on. It is noticeable enough that when I'm using the web admin to change between a few settings I'll quite often see it. Most of the time it doesn't cause any problems as it always finishes processing after the delay. Every now and then though it doesn't come back. ASSP won't respond to any shutdown commands so I have to kill the process, remove the pid file and start it back up manually. My monitoring scripts only kick in if they can't connect to port 25/55555. I suspect this won't help as that doesn't look to be performance related and All the best, Colin Waring. -----Original Message----- From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] Sent: 09 March 2015 05:38 To: ASSP development mailing list Subject: Re: [Assp-test] fixes in assp 2.4.4 build 15067 Colin, >If I understand more of how experimental this is and the next step is >to HA the database At this time the code is very experimental and very special. It aims to fix SMTP performance problems for an ISP, which holds around 20.000 domains. The concept of the central RDB (for HMM and Bayesian) backend is not fast enough to process several hundred thousands or million mails a day. If 100.000 mails have to be processed with HMM and/or Bayesian in a day, this will lead in to 6.000.000 - 60.000.000 SQL queries a day (only for HMM....). What DB engine (cluster) is able to do this? And this is only the average calculation - what about the peaks? The code is currently specialized for the environment of this single ISP and is not generic enough to go public. There are currently no changes made to enhance the implementation of other features, like blockreporting or anything else. Thomas Von: Colin Waring <co...@dolphinict.co.uk> An: ASSP development mailing list <assp-test@lists.sourceforge.net> Datum: 08.03.2015 15:18 Betreff: Re: [Assp-test] fixes in assp 2.4.4 build 15067 Hi Thomas, I'd be very interested to know more details on the ultimate aim with the ISP option. I support the idea of subscription for the higher end as it will help create funding for you past donations. Is the aim of the addition to add support for extended scalability or do you have ideas for the future to make additional features available? If you remember we exchanged emails a while back about some of the features that I could see benefiting a larger setup and we are looking into how to implement things at the moment. I've already implemented clustered file systems and the next step is to HA the database. The biggest concern for me in scaling up is the block reports being generated on each server individually. If I understand more of how experimental this is and what could go wrong then I may be able to help with testibg. All the best, Colin Waring On 8 Mar 2015 12:39, Thomas Eckardt <thomas.ecka...@thockar.com> wrote: Hi all, fixed in assp 2.4.4 build 15067: - on some windows systems 'Win32::Unicode' was detected as unavailable, even it was correctly installed - the alpha index was not working in build 15059 - HMM was not working, if 'spamdb' was set to a plain file, placed in a subfolder like: db/spamdb added: - This build contains experimental code to setup assp in very large ISP environments, with a very high workload caused by HMM, Bayesian and DNS. Such a setup requires an enormous and expensive amount of hardware resources, a very high knowledge in system design and OS scripting. minimum requirements: - assp: 64Bit OS, all SSD, 16GB RAM, 8 CPU cores, 64Bit Perl (multiple larger systems expected) - external high available enterprise database server - high available and very fast DNS-servers This ISP setup option is subject to become a payed licensed feature. Thomas DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
