Re: sysctl & process limitations
On Tue, 5 Feb 2002, Manuel Bouyer wrote: > On Tue, Feb 05, 2002 at 07:59:20AM -0700, Randy Arabie wrote: > > On Mon, 4 Feb 2002, Randy Arabie wrote: > > > > > Right. Like this: > > > > > > mysql -u someuser -p -D somedatabase < ./data.sql > > > > > > No. I didn't raise the shell limits, I rose the kernel maxfiles and > > > the mysqld process proc.pid.rlimit.descriptors.soft limit. > > > > > > I'll try the shell tonight. > > > > Summary of Last Nights activities: > > > > 1) Switched back to the GENERIC kernel. > > 2) Edit /etc/sysctl.conf; added a line kern.maxfiles=5000 > > 3) Reboot > > 4) Bump hard and soft proc.[mysqld_pid].rlimit.descriptors up to 2500. > > 5) Bump hard and soft proc.[shell_pid].rlimit.descriptors up to 2500. > > 6) Attempt to load the data, failure once again, same error message: > > > > "ERROR 1017 at line 445: Can't find file: './geeklog/userinfo.frm' (errno: 23)" > > > > 7) fstat | wc -l shows 203 open files (done immediately following step 6 above. > > 8) fstat | grep mysqld | wc -l shows mysqld only has 64 open files. > > 9) The database only has 38 tables, and the datafile I'm trying to load is only > > 37 KB. > > > > Someone suggested a few days ago that perhaps the error reported is not accurate. >How would > > I test that? > > > > My 'gut' says this is probably an easily resolved configuration issue, but I'm >stumped on > > where to go next. > > Now I would try to ktrace both mysqld and the mysql loading the > tables ... You mention that, mysqld AND mysql. You will note above that when I checked mysqld with fstat, there were only 64 files open. That, coincidentally, is the default proc.curproc.rlimit.descriptors.soft setting. Does mysqld fork they mysql process, which may not be inheriting the limits set for mysqld? Can I add a line in /etc/sysctl.conf to allow all processes to have a greater default soft limit for descriptors? Or could I recompile my kernel to change that? I looked in kernel config and didn't see the options to change such things. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
> On Mon, 4 Feb 2002, Manuel Bouyer wrote: > > > What a minute ... you're loading the tables to a 'mysqld' process, using > > the 'mysql' command, rigth ? > > Maybe it's just the mysql command which runs out of files descriptors ? > > Did you try to raise the limit of the shell before starting it ? On Mon, 4 Feb 2002, Randy Arabie wrote: > Right. Like this: > > mysql -u someuser -p -D somedatabase < ./data.sql > > No. I didn't raise the shell limits, I rose the kernel maxfiles and > the mysqld process proc.pid.rlimit.descriptors.soft limit. > > I'll try the shell tonight. Summary of Last Nights activities: 1) Switched back to the GENERIC kernel. 2) Edit /etc/sysctl.conf; added a line kern.maxfiles=5000 3) Reboot 4) Bump hard and soft proc.[mysqld_pid].rlimit.descriptors up to 2500. 5) Bump hard and soft proc.[shell_pid].rlimit.descriptors up to 2500. 6) Attempt to load the data, failure once again, same error message: "ERROR 1017 at line 445: Can't find file: './geeklog/userinfo.frm' (errno: 23)" 7) fstat | wc -l shows 203 open files (done immediately following step 6 above. 8) fstat | grep mysqld | wc -l shows mysqld only has 64 open files. 9) The database only has 38 tables, and the datafile I'm trying to load is only 37 KB. Someone suggested a few days ago that perhaps the error reported is not accurate. How would I test that? My 'gut' says this is probably an easily resolved configuration issue, but I'm stumped on where to go next. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Mon, 4 Feb 2002, Manuel Bouyer wrote: > What a minute ... you're loading the tables to a 'mysqld' process, using > the 'mysql' command, rigth ? > Maybe it's just the mysql command which runs out of files descriptors ? > Did you try to raise the limit of the shell before starting it ? Right. Like this: mysql -u someuser -p -D somedatabase < ./data.sql No. I didn't raise the shell limits, I rose the kernel maxfiles and the mysqld process proc.pid.rlimit.descriptors.soft limit. I'll try the shell tonight. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Mon, 4 Feb 2002, William R. Mussatto wrote: > Have you checked on the ulimit for the user that is mysql? I did, but had only set it "dynamically". I did raise it quite high, 5000 I believe. After that I initiated the load, but from a "normal" user shell which had the default proc.[shell_pid].rlimit.descriptors.soft=64 limitation. Hence my question, from where does the load inherit its limit. I'm stepping back and looking at where I should set it at the kernel level, and then where I can set it for specific users and/or processes. -- Cheers! Randy ======== Randy Arabie GnuPG Key Info -- Fingerprint: 7E25 DFA2 EF72 9551 9C6C 8AA6 6E8C A0F5 7E33 D981 Key ID: 7C603AEF http://www.arabie.org/keys/rrarabie.gnupg - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Mon, 4 Feb 2002, William R. Mussatto wrote: > Have you checked on the ulimit for the user that is mysql? I've checked a lot of things. Which would apply: kern.maxfiles proc.[mysqld_pid].rlimit.descriptors.soft proc.[shell_pid].rlimit.descriptors.soft Obviously, the process rlimits cannot exceed the limits set in the kernel. But, between the other two, which one takes precedence...the shell from which the command was invoked or the actual mysqld process? -- Cheers! Randy ==== Randy Arabie GnuPG Key Info -- Fingerprint: 7E25 DFA2 EF72 9551 9C6C 8AA6 6E8C A0F5 7E33 D981 Key ID: 7C603AEF http://www.arabie.org/keys/rrarabie.gnupg - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Sat, 2 Feb 2002, Dan Nelson wrote: > In the last episode (Feb 02), Randy Arabie said: > > On Sat, 2 Feb 2002, Dan Nelson wrote: > > mysql. > > > > kern.maxfiles = 1772 > > > > > > Raise that to (say) 5000. I assume netbsd has an /etc/sysctl.conf you > > > can use, or you can do it manually via "sysctl -w kern.maxfiles=5000" . > > > > How high can I go?? > > > > Tried 5000, and it still bombs at the same point. I set that at the > > kernel level and the descriptors at the process level. > > The error might not actually be the kernel limit being reached. It > might just be the per-process rlimit. What does "ulimit -n" print? I don't think so. I raised the limit for mysqld descriptors to 5000. # sysctl -w proc.220.rlimit.descriptors.hard=unlimited # sysctl -w proc.220.rlimit.descriptors.soft=unlimited The kernel maxfiles was already set to 5000. Those commands raised the limit for process 220 (mysqld) to 5000. And, the load still fails with Errcode: 23. -- Cheers! Randy Randy Arabie GnuPG Key Info -- Fingerprint: 7E25 DFA2 EF72 9551 9C6C 8AA6 6E8C A0F5 7E33 D981 Key ID: 7C603AEF http://www.arabie.org/keys/rrarabie.gnupg - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Sat, 2 Feb 2002, Dan Nelson wrote: > > kern.maxfiles = 1772 > > Raise that to (say) 5000. I assume netbsd has an /etc/sysctl.conf you > can use, or you can do it manually via "sysctl -w kern.maxfiles=5000" . How high can I go?? Tried 5000, and it still bombs at the same point. I set that at the kernel level and the descriptors at the process level. Frustrating...it isn't a big mySQL database. Only 38 tables, and presently they are empty. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Sat, 2 Feb 2002, Manuel Bouyer wrote: > On Sat, Feb 02, 2002 at 10:33:21AM -0700, Randy Arabie wrote: > > > > > You may also want to ktrace mysqld while trying to load the > > > database to get more details. > > > > Tried this. I'm not much on analyzing a kernel trace, but I could see > > from the trace that the file was opened. And, it appears that some of > > the data was inserted. However, I can't check via the database. When I > > log in and try "desc tablename" I get the same Errcode: 23 for the > > corresponding mysql *.MYD file. This happens with a number of the tables > > in the database. > > And how much files does mysqld have ? > fstat |grep mysqld |wc -l Seven (7). I just shut down sendmail & httpd. Now "# fstat | wc -l" returns 92. I still have the same problem loading the data. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: sysctl & process limitations
On Sat, 2 Feb 2002, Manuel Bouyer wrote: > On Sat, Feb 02, 2002 at 08:23:49AM -0700, Randy Arabie wrote: > > Environment:AlphaStation 255 w/ 64 MB RAM > > NetBSD 1.5.2 - custom kernel > > MySQL 3.23.47 > > > > I'm trying to load data into tables in a mysql database. When I feed the > > load script into the database it fails about 3/4 through. Error follows: > > > > > > ERROR 1105 at line 11: File ./geeklog/userinfo.MYD not found (Errcode: 23) > > > > 23 is "Too many open files in system" > > Maybe the whole system has too many open files ? > The limit is kern.maxfiles and you can count them with 'fstat'. The output of "# fstat | wc -l" is 213. kern.maxfiles = 1772 > You may also want to ktrace mysqld while trying to load the > database to get more details. Tried this. I'm not much on analyzing a kernel trace, but I could see from the trace that the file was opened. And, it appears that some of the data was inserted. However, I can't check via the database. When I log in and try "desc tablename" I get the same Errcode: 23 for the corresponding mysql *.MYD file. This happens with a number of the tables in the database. -- Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
sysctl & process limitations
Environment:AlphaStation 255 w/ 64 MB RAM NetBSD 1.5.2 - custom kernel MySQL 3.23.47 I'm trying to load data into tables in a mysql database. When I feed the load script into the database it fails about 3/4 through. Error follows: ERROR 1105 at line 11: File ./geeklog/userinfo.MYD not found (Errcode: 23) Looking at a few newsgroup postings, I beleive this is due to resource limitations. Specifically, I may have reached the max open descriptors limit. I've used the sysctl command to set all soft limits to unlimited, and still get the error. Here is the output of sysctl for my mysqld process: proc.4012.rlimit.cputime.soft = unlimited proc.4012.rlimit.cputime.hard = unlimited proc.4012.rlimit.filesize.soft = unlimited proc.4012.rlimit.filesize.hard = unlimited proc.4012.rlimit.datasize.soft = 1073741824 proc.4012.rlimit.datasize.hard = 1073741824 proc.4012.rlimit.stacksize.soft = 33554432 proc.4012.rlimit.stacksize.hard = 33554432 proc.4012.rlimit.coredumpsize.soft = unlimited proc.4012.rlimit.coredumpsize.hard = unlimited proc.4012.rlimit.memoryuse.soft = 57597952 proc.4012.rlimit.memoryuse.hard = 57597952 proc.4012.rlimit.memorylocked.soft = 57597952 proc.4012.rlimit.memorylocked.hard = 57597952 proc.4012.rlimit.maxproc.soft = 532 proc.4012.rlimit.maxproc.hard = 532 proc.4012.rlimit.descriptors.soft = 1772 proc.4012.rlimit.descriptors.hard = 1772 According to this, I should be able to have 1772 files open...correct? I'm only trying to load data into 38 tables. And, IIRC, mysql only opens two files per table. So, I should not be encountering my descriptors limit. Any advice would be appreciated. TIA Cheers! Randy - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail <[EMAIL PROTECTED]> To unsubscribe, e-mail <[EMAIL PROTECTED]> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
