-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Brian,
On 1/25/13 9:00 PM, Brian Braun wrote:
> Until recently, all this happened in a syncronous way. I mean,
> before the Tomcat thread delivered the response, it had to create
> the records in the database table. The problem is that this log
> used to take more than 90% of the total time, and even worse: when
> the database got slower then the service took about 10-15 seconds
> instead of just 20 milliseconds.
Ouch.
What kind of tables are you using? How many records count as a "log"?
In how many tables? How many indexes do you have on that/those table/s?
> I recetly made an improvement: Each Tomcat thread creates an extra
> thread doing a "(new Thread(new certain Object)).start();" that
> takes care of the SQL insertion in an asyncronous way, so the
> response gets to the clients faster. But these threads take too
> much RAM when MySQL runs slower and threads multiply, and with a
> few thousands of them the JVM Tomcat runs of the memory.
Yup: you'll at least want to run an Executor to limit the number of
new threads you create. That's a bad suggestion, anyway, because there
are better techniques you can use.
> What I need is to be able to accept as much HTTP requests as
> possible, to log every one of them as fast as possible (not
> syncronously), and to make everything fast and with a very low
> usage of RAM when MySQL gets slow and inserts need to queue. I
> think I need some kind of queue to buffer the entries when the
> speed of http request is higher than the speed of insertions in the
> database log.
So, if you don't need synchronous log-writing does that mean you can
also tolerate some logs being missed? They are kind of two sides of
the came coin.
> I'm thinking about these ideas:
>
> 1- Creating some kind of FIFO queue myself, maybe using some of
> those Apache commons collections, and the some kind of thread that
> polls the collection and creates the database records. But what
> collection should I use? And how should I program the thread that
> polls it, so it won't monopolize the CPU? I think that a "Do while
> (true)...." would eat the CPU cycles. And that about making it
> thread safe? How to do it?
You can use while(!stop) { Object.wait(); dequeue(); } which will be
somewhat reasonable. Then you set "stop" when you want to shut-down
your queue-processor (like from a ServletContextListener). Be sure to
make that member volatile.
> 2- log4J? I have never used it directly, but it seems that this
> framework is algo designed to creat "appenders" that talk to the
> database. Would that be the way to do it?
If you can deal with the restrictions of log4j (basically, you
generate a line of text and that can get inserted into the db). I
haven't checked how the JDBCAppender works, but I would hope they
write more than one message to the db in each transaction.
Honestly, I would try log4j: it's already written for you and you can
log to it very simply and configure it simply as well. So, it's cheap
to compare it to your current situation.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlEGlIIACgkQ9CaO5/Lv0PAEsACfVvL/sAxua5yYF2UvARPzt4jS
M6EAoIJ71sbsYQmUwHmfqWKCoLDFYCwS
=LlXJ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
