Just getting that number of processes running I think would be a challenge. A setup I recently worked on runs a few hundred
processes per box, and that kind of maxes out the CPU.
Approach 1, been there, done that. Too messy.
Approach 2, considered it, but you may end up with processes that never connect. You would need a queueing/scheduling mechanism.
Essentially you would be trying to do what an OS does, manage resources to make sure every process gets it's turn.
Approach 3, what we currently use. The processes connect to the db, does a bulk insert and then disconnects. We decided to limit
each process to blocks of 100. Inserting a single record at a time will quickly degrade. This setup actually moved the bottleneck
from the database to the processes doing their job. When each process starts, it inserts a record into a table and gets it's id. The
process then handles the autoincrement value. The unique id for each record is then the process "id" plus the increment value.
To really scale, you may want to look into the black hole table format. Essentially it's a black hole, nothing is saved so there
really isn't much overhead. But you set it up to be replicated and a replication log is generated. An easy setup would be to have
multiple tables on a "master" server, each table replicating a black hole table from another server. Then create a merge table
encompassing the multiple tables for easy querying.
This is the next idea we are pursueing, so it may or may not work.
----- Original Message -----
From: "bruce" <[EMAIL PROTECTED]>
To: <mysql@lists.mysql.com>
Sent: Friday, August 25, 2006 1:12 PM
Subject: file i/o operations...
hi...
i'm trying to determine which is the better way/approach to go. should an
app do a great deal of file i/o, or should it do a great deal of read/writes
to a mysql db...
my test app will create a number of spawned child processes, 1000's of
simultaneous processes, and each child process will create data. the data
will ultimately need to be inserted into a db.
Approach 1
-----------
if i have each child app write to a file, i'm going to have a serious hit on
the disk, for the file i/o, but i'm pretty sure Centos/RH could handle it.
(although, to be honest, i don't know if there's a limit to the number of
simultaneous file descriptors that the OS allows to be open at the same
time.) i'm assuming that the number is multiples of magnitudes more than the
number of simultaneous connections i can have with a db....
i could then have a process/app collect the information from each output
file, writing the information to the db, and deleting the output files as
required.
Approach 2
----------
i could have each child app write to a local db, with each child app,
waiting to get the next open db connection. this is limited, as i'd run into
the max connection limit for the db. i'd also have to implement a process to
get the information from the local db, to the master db. ..
Approach 3
-----------
i could have each child app write directly to the db.. the problem with this
approach is that the db has a max regarding the number of simultaneous
connections, based on system resources. this would be the cleanest
solution..
so... anybody have any thoughts/comments as to how one can essentially
accept 1000's-10000's of simultaneous hits with an app...
i've been trying to find out if there's any kind of distributed
parent/child/tiered kind of app, where information/data is more or less
collected and received at the node level...
does anyone know of a way to create a distributed kind of "db" app, where i
can enter information into a db on a given server, and the information is
essentially pulled into the master server from the child server...
thanks
-bruce
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
