MySQL Error: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

2008-09-23 Thread Sam Nilsson

Hello,

The error message: Can't create a new thread (errno 35); if you are not 
out of available memory, you can consult the manual for a possible 
OS-dependent bug


Our website started getting this error several weeks ago (when we 
increased the number of application server machines connecting to the 
database from 5 to 7.  It seems to happen anywhere from every 2 days to 
once a week, there does not seem to be a consistent pattern.  Our CPU 
utilization, memory utilization,  and number of HTTP Requests seem to be 
at "normal" (they are no where closed to being maxed out).  When this 
error happens we CAN log onto the server with the MySQL database, but I 
CAN NOT use the mysql client to connect to the mysql database (localhost 
connection as root) - it reports the error mentioned above.   The way we 
temporarily solve the problem is to restart one of our web servers.


Here is our technology stack:

Web Servers: 7, Each server has:
  4 Gb of Memory on each server
  FreeBSD 6.2-RELEASE-p9
  2.4 GHz CPU
  Apache 2.2
  Webware for Python
  Python 2.5

DB Servers: One Master, Two Read Only (replication)
  4 GB of Memory on each server
  FreeBSD 6.3-RELEASE-p3 FreeBSD 6.3-RELEASE-p3
  MySQL 5.0.1


We have tried bumping up our number of max allowed connections (up to 
1000) on MySQL and kern.ssize to 512Mb for FreeBSD.   That did not help. 
 The max connections usually hovers ~300, so it does not even get close 
to the 1000 we have set.  Finally the last place that we have looked is 
the openfiles, it is set at 20,000 for the whole system, with 14K per 
process on the FreeBSD operating system.



Here are some relevent items from my.cnf:
 - set-variable = max_connections=1000
 - set-variable = key_buffer_size=384M
 - set-variable = read_buffer_size=64M
 - set-variable = read_rnd_buffer_size=32M
 - set-variable = thread_cache_size=20

Any help would be much appreciated.

Thanks
- Sam Nilsson


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: MySQL Error: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

2008-09-23 Thread Vincent Hoffman
Sam Nilsson wrote:
> Hello,
>
> The error message: Can't create a new thread (errno 35); if you are
> not out of available memory, you can consult the manual for a possible
> OS-dependent bug
>
> Our website started getting this error several weeks ago (when we
> increased the number of application server machines connecting to the
> database from 5 to 7.  It seems to happen anywhere from every 2 days
> to once a week, there does not seem to be a consistent pattern.  Our
> CPU utilization, memory utilization,  and number of HTTP Requests seem
> to be at "normal" (they are no where closed to being maxed out).  When
> this error happens we CAN log onto the server with the MySQL database,
> but I CAN NOT use the mysql client to connect to the mysql database
> (localhost connection as root) - it reports the error mentioned
> above.   The way we temporarily solve the problem is to restart one of
> our web servers.
>
> Here is our technology stack:
>
> Web Servers: 7, Each server has:
>   4 Gb of Memory on each server
>   FreeBSD 6.2-RELEASE-p9
>   2.4 GHz CPU
>   Apache 2.2
>   Webware for Python
>   Python 2.5
>
> DB Servers: One Master, Two Read Only (replication)
>   4 GB of Memory on each server
>   FreeBSD 6.3-RELEASE-p3 FreeBSD 6.3-RELEASE-p3
>   MySQL 5.0.1
>
>
> We have tried bumping up our number of max allowed connections (up to
> 1000) on MySQL and kern.ssize to 512Mb for FreeBSD.   That did not
> help.  The max connections usually hovers ~300, so it does not even
> get close to the 1000 we have set.  Finally the last place that we
> have looked is the openfiles, it is set at 20,000 for the whole
> system, with 14K per process on the FreeBSD operating system.
>
>
> Here are some relevent items from my.cnf:
>  - set-variable = max_connections=1000
>  - set-variable = key_buffer_size=384M
>  - set-variable = read_buffer_size=64M
>  - set-variable = read_rnd_buffer_size=32M
>  - set-variable = thread_cache_size=20
>
> Any help would be much appreciated.
>
> Thanks
> - Sam Nilsson
>
If you havent already, you could try increasing the per process memory
limit as per examples in
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/hackers/2008-05/msg00258.html
(man tuning also says a bit about these tuneables but doesnt have the
examples that post does)


Vince
>
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "[EMAIL PROTECTED]"

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: MySQL Error: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

2008-09-23 Thread Mel
On Tuesday 23 September 2008 12:13:58 Sam Nilsson wrote:

> DB Servers: One Master, Two Read Only (replication)
>4 GB of Memory on each server
>FreeBSD 6.3-RELEASE-p3 FreeBSD 6.3-RELEASE-p3
>MySQL 5.0.1

> Here are some relevent items from my.cnf:
>   - set-variable = max_connections=1000
>   - set-variable = key_buffer_size=384M
>   - set-variable = read_buffer_size=64M
>   - set-variable = read_rnd_buffer_size=32M
>   - set-variable = thread_cache_size=20

You're shooting yourself in the foot:
1000*2MB=2G for thread stack
+ 384M
+ 1000 * (sort_buffer_size+64M+binlog_cache_size_innodb)

You don't have that much memory.
http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

There's a similar formula for MyISAM, but can't seem to find it at the moment.


-- 
Mel

Problem with today's modular software: they start with the modules
and never get to the software part.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: MySQL Error: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

2008-09-23 Thread Sam Nilsson

Vincent Hoffman wrote:

If you havent already, you could try increasing the per process memory
limit as per examples in
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/hackers/2008-05/msg00258.html
(man tuning also says a bit about these tuneables but doesnt have the
examples that post does)


Vince


Hi Vince,

Thanks for the advice. We have already raised the memory limits:

kern.maxdsiz="1843M" # 1.8GB
kern.dfldsiz="1843M" # 1.8GB

Any other ideas?

- Sam
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"