On Friday 06 December 2013 09:35 PM, Jonathan Vanasco wrote:
[...]
It sounds to me like you have an anti-pattern somewhere. Your entire
application shouldn't hang because of a database connectivity issue.
Specific requests to code that is trying to interact with the database
should hang -- but the application shouldn't itself. SqlAlchemy
shouldn't even connect to the database on a request that doesn't have DB
logic in it.
Sorry, I forgot to mention that all my application's requests require
database queries. As expected, other parts of my application are indeed
working properly. In fact, as mentioned, I was able to connect to the
gevent backdoor (like twisted manhole) port and obtain the provided
trace on the 'hung' server.
The immediate concern I see with your fix -- at least in your situation
-- is that it could create a further surge and backlog on the MySQL
server. I've worked with a few large online properties where some sort
of surge like this ( where a bunch of app-servers kept reconnecting to
mysql in a short time ) ended up creating a self-inflicted Denial of
Service that took the sites offline, and in most of the instances the DB
got corrupted and had to be repaired.
Hundreds of connections per second to the database are being created by
a legacy PHP application. My Python/SQLAlchemy application is much more
nice behaving during connection pooling. It just got caught in the
overall mess.
My database is able to handle lot higher connections than the temporary
backlog surge. The proper fix for the backlog problem, of course, is to
increase the backlog by setting MySQL backlog parameter and overall
system backlog values in the kernel. I have fixed this [1].
However, the current problem is that once a temporary backlog overflow
is hit (or a few connections are lost for some reason), the application
does not *ever* serve another database related request. It certainly
requires the fix that has been committed.
Links:
1)
http://www.mysqlperformanceblog.com/2012/01/06/mysql-high-number-connections-per-secon/
Thank you,
--
Sunil
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
