>1) Why InnoDB? Does rollback and integrity matter a ton? If not, there
are a few other engines that may be worth looking at for pure speed and
also not using up so much disk space as what innoDB does.
Yes, rollback matters a ton! Data integrety is very important for ASSP.
Diskspace is never a problem!
2) Why does it need 1000 connections? To build out MySql that can handle
1000 simultaneous connections requires significant hardware. At least, in
my experience, that has been the case.
Every thread in V2 is running completely independend. So every thread
needs its own connection to every DB table.
Significant Hardware is not needed. ASSP only holds the connection opened
to have fast access to the data. A DB request is only done if a hash s
accessed by the code.
The calculation of the needed number of connections is very simple (
(SMTP-Workers + 3) * 22 )
MySQL could be used, but it is not my first choice.
>Maybe ASSP should move all datastores to sqlite? It is easy to
implement, near no changes in syntax, very lightweight, and very good at
performance. >It is also portable and easy to maintain. No database need
be installed, it pretty much just sits in disk as a binary file where you
tell it to be.
BerkeleyDB is implemented and could be used for every hash - this is the
best option for single systems, if you have no database knowledge.
If you have multiple systems sharing the same data, you'll need a DB
engine with a replication mechanism - and the first choice for this are
DB/2, oracle and on windows - MSSQL. Professional users will already have
such a DB and for this reason, this (should be) is supported by ASSP.
>This could potentially remove all the code and overheard in ASSP for all
the other database connection systems, and make ASSP fully self contained,
>and probably remove several memory chewing perl modules for just one.
There is not so much code to remove in that case (less than 100 lines).
There is no special code in assp for any DB (except BerkeleyDB for the
init). All DB's using the same code - but there own DB driver. Any other
DB will not use more or less memory inside ASSP than sqlite.
>This is interesting ( http://www.perlmonks.org/?node_id=152749 )
>According to that, BDB can be faster, or equal sqlite, and somewhat
slower than sqlite2. But he was using committed inserts, which I have
never used >in heavy production on sqlite.
There is no faster hash based DB on that planet than BerkeleyDB.
The interesting test (the first) is completly wrong. It uses different
mechanism to access the DB's and compares the benchmark. BDB and CDB uses
a tie - interface not the native access methodes, which produces a large
overhead.
The last topic at this page shows a real relation:
Benchmark: timing 30000 iterations of
S B+Tree, S B+Tree(A), S Hash, S Hash(A), S SQLite, S
SQLite 2...
S B+Tree: 1 wallclock secs ( 0.77 usr + 0.09 sys = 0.86 CPU) @
34965.03/s (n=30000)
S B+Tree(A): 1 wallclock secs ( 0.95 usr + 0.05 sys = 1.00 CPU) @
30000.00/s (n=30000)
S Hash: 2 wallclock secs ( 1.03 usr + 1.63 sys = 2.66 CPU) @
11295.18/s (n=30000)
S Hash(A): 3 wallclock secs ( 1.47 usr + 1.36 sys = 2.83 CPU) @
10608.20/s (n=30000)
S SQLite: 11 wallclock secs ( 6.80 usr + 3.69 sys = 10.49 CPU) @
2861.23/s (n=30000)
S SQLite 2: 4 wallclock secs ( 3.89 usr + 0.09 sys = 3.98 CPU) @
7530.12/s (n=30000)
Thomas
Von: Scott Haneda <talkli...@newgeo.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum: 14.03.2010 01:59
Betreff: Re: [Assp-test] Assp dying
On Mar 13, 2010, at 12:00 AM, Thomas Eckardt wrote:
> If you run a DNS-server on the same system calculate 1/2 CPU core for
this
> process - 275,000 connections per day , at minimum ~ 5-6 DNS-querys per
> connection => 1.500.000 DNS querys per day or ~16 per second .
> If you run a DB-server on the same system calculate 1.5 CPU cores for
this
> engine and depending on the used engine 300-600 MB RAM. 275,000 * 100 DB
> requests per connection are 27.500.000 DB requests per day or ~320 per
> second
> If you want to use ClamAV - try it, but I think it will use too much CPU
> in any case on your system !
Are you statin that for the DNS alone you should prepare 1/2 a core of CPU
for those lookups? Is this based on Windows DNS resolvers, or linux? ~16
q/s does not seem like a lot. I current DNS which uses named, handles
well more than that every day, according to `ps`
CPU %mem VSZ
named 4 0.0 0.6 ?? Ss 3Mar10 13:28.29 /usr/sbin/named -
Looks like it will not even register that memory usage, and CPU usage is
very low down there. Looking at my query.log, it appears right now, this
instant, I just had 58 AAAA or A record lookups over the time of
13-Mar-2010 16:39:01.020 to 13-Mar-2010 16:39:01.914. Maybe different DNS
servers on different platforms can handle this much better?
This is a wimpy old machine too :)
Processor Name: PowerPC G4 (3.3)
Processor Speed: 1.25 GHz
I have 4 slots only for memory, all filled identically ( total ~2GB ):
Size: 512 MB
Type: DDR SDRAM
Speed: PC2600U-25330
> Install MySQL 5.x , create a DB (eg) 'assp' , use the InnoDB engine. ,
> configure MySQL to enable at least 1000 connections
I may have to look into this. A few questions:
1) Why InnoDB? Does rollback and integrity matter a ton? If not, there
are a few other engines that may be worth looking at for pure speed and
also not using up so much disk space as what innoDB does.
2) Why does it need 1000 connections? To build out MySql that can handle
1000 simultaneous connections requires significant hardware. At least, in
my experience, that has been the case.
Maybe ASSP should move all datastores to sqlite? It is easy to implement,
near no changes in syntax, very lightweight, and very good at performance.
It is also portable and easy to maintain. No database need be installed,
it pretty much just sits in disk as a binary file where you tell it to be.
This could potentially remove all the code and overheard in ASSP for all
the other database connection systems, and make ASSP fully self contained,
and probably remove several memory chewing perl modules for just one.
This is interesting ( http://www.perlmonks.org/?node_id=152749 )
According to that, BDB can be faster, or equal sqlite, and somewhat slower
than sqlite2. But he was using committed inserts, which I have never used
in heavy production on sqlite.
I think it all comes down to transaction size, but it something that
should be looked into, as a tuning mechanism. One will work best for ASSP
since ASSP does very specialized work with the database.
> To minimize the workload of your system (DNS, DB), try to block
> connections as early as possible (helo, IP, PTR, MX...) - content based
> blocking (bayes, bombs ...) should be minimized.
Very good advice. I find you can reduce load on any system, be it ASSP,
or any other mail system, but applying these techniques. There is so
little overhead to blocking early in the connection phase, especially
ehlo/helo based blocking. One one of my servers I would guess I am
blocking 90% of bots with one rule of ehlo/helo does not contain a dot.
This requires almost no CPU at all to do.
Thank you for the information Thomas, good to hear a run down of how ASSP
should be looked at being set up for medium to large email systems. Thank
you.
--
Scott * If you contact me off list replace talklists@ with scott@ *
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
