Re: Fw: Problems with high-use tables
My problems seem to be occuring with the table receiving the processed firewall logs. However I don't understand this - there may be an insert of 200-500 records every five minutes, but reporting and cleanup only touches this table once a day. My syslog table has inserts and deletes run on it all day long. When rebuilding my server, I dumped the table structure and rebuilt the database. However, I did not convert the data, so everything in this DB is no more than a month old (summary data). -Chris - Original Message - From: Dan Vande More [EMAIL PROTECTED] To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, July 19, 2002 9:14 PM Subject: RE: Fw: Problems with high-use tables Chris, I built an exe in perl that runs on windows via a logon script that stores the username and whether they have (win)vnc or not to a mysql table. My pix firewall syslogs (Through a fifo on a linux box, perl does dbd to mysql) all url access to another mysql table that holds a timestamp, user, and url accessed. My table currently has 380,000 records, dating back to Nov 20th. Sometimes there can be as many as 12 or 14 inserts a second, and on weekends, 5 a day. I run 3.23.49 so I can vouch for being able to handle data that big at those rates. The question I have is which data files are getting corrupted? The ones straight from the firewall? The ones that just got parsed by vb or the mysql ones? After you rebuilt your server, did you just copy the tables over (ie the actual MYD, MYI and frm files) or did you dump from and dump to? If you did just copy the files, did you copy them between different (mysql)builds/os/hardware platforms? What version are you running now? IIRC some people on this list said that 3.23.51 was kind of buggy and exhibited several weird problems (I think, please confirm). If you need some perl DBD stuff for mysql, I will share it with you, though I wrote this for a PIX firewall, and there is some source code I cannot find (I compiled the exe with perl2exe). I don't know much of perl, and this was my first (and only so far) script, but I'll share it with you if you wish. Thanks Dan -Original Message- From: Christopher A. Libby [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 18, 2002 11:49 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Fw: Problems with high-use tables When the scripts where originally written, I didn't have a clue about the DBD::MySQL module. I've also got scripts running in VBScript via ODBC, so I wanted to stick to a standard I/O method. -Chris - Original Message - From: William R. Mussatto [EMAIL PROTECTED] To: Christopher A. Libby [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Thursday, July 18, 2002 11:30 AM Subject: Re: Fw: Problems with high-use tables Just a quick question. Why are you not running DBD::Mysql directly? It should eliminate the ODBC layer and help with speed. On Thu, 18 Jul 2002, Christopher A. Libby wrote: Date: Thu, 18 Jul 2002 08:22:33 -0400 From: Christopher A. Libby [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Fw: Problems with high-use tables - Original Message - From: Christopher A. Libby [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, July 16, 2002 3:32 PM Subject: Problems with high-use tables Hi - I've been using MySQL for a variety of functions with great success over the last year. However, I recently rebuilt my log database server after a crash, and I've been having a lot of trouble with two tables getting corrupted. One table stores my syslog entries from Kiwi Syslod Daemon, and the other stores the actual firewall logs extracted from the syslog entries. Every five minutes I run a process which reads in all the syslog entries, extracts the firewall logs from them, then deletes the entries from the syslog table. My problem is I keep getting corrupted data files, which brings the whole system to a halt. I'm running MySQL v3 (latest build) on a WinNT4 box. I'm using Perl MyODBC to process the data. Any thoughts? -Chris - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php Sincerely, William Mussatto, Senior Systems Engineer CyberStrategies, Inc ph. 909-920-9154 ext. 27 - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php - Before posting, please
Fw: Problems with high-use tables
- Original Message - From: Christopher A. Libby [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, July 16, 2002 3:32 PM Subject: Problems with high-use tables Hi - I've been using MySQL for a variety of functions with great success over the last year. However, I recently rebuilt my log database server after a crash, and I've been having a lot of trouble with two tables getting corrupted. One table stores my syslog entries from Kiwi Syslod Daemon, and the other stores the actual firewall logs extracted from the syslog entries. Every five minutes I run a process which reads in all the syslog entries, extracts the firewall logs from them, then deletes the entries from the syslog table. My problem is I keep getting corrupted data files, which brings the whole system to a halt. I'm running MySQL v3 (latest build) on a WinNT4 box. I'm using Perl MyODBC to process the data. Any thoughts? -Chris - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Re: Fw: Problems with high-use tables
When the scripts where originally written, I didn't have a clue about the DBD::MySQL module. I've also got scripts running in VBScript via ODBC, so I wanted to stick to a standard I/O method. -Chris - Original Message - From: William R. Mussatto [EMAIL PROTECTED] To: Christopher A. Libby [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Thursday, July 18, 2002 11:30 AM Subject: Re: Fw: Problems with high-use tables Just a quick question. Why are you not running DBD::Mysql directly? It should eliminate the ODBC layer and help with speed. On Thu, 18 Jul 2002, Christopher A. Libby wrote: Date: Thu, 18 Jul 2002 08:22:33 -0400 From: Christopher A. Libby [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Fw: Problems with high-use tables - Original Message - From: Christopher A. Libby [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, July 16, 2002 3:32 PM Subject: Problems with high-use tables Hi - I've been using MySQL for a variety of functions with great success over the last year. However, I recently rebuilt my log database server after a crash, and I've been having a lot of trouble with two tables getting corrupted. One table stores my syslog entries from Kiwi Syslod Daemon, and the other stores the actual firewall logs extracted from the syslog entries. Every five minutes I run a process which reads in all the syslog entries, extracts the firewall logs from them, then deletes the entries from the syslog table. My problem is I keep getting corrupted data files, which brings the whole system to a halt. I'm running MySQL v3 (latest build) on a WinNT4 box. I'm using Perl MyODBC to process the data. Any thoughts? -Chris - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php Sincerely, William Mussatto, Senior Systems Engineer CyberStrategies, Inc ph. 909-920-9154 ext. 27 - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
Problems with high-use tables
Hi - I've been using MySQL for a variety of functions with great success over the last year. However, I recently rebuilt my log database server after a crash, and I've been having a lot of trouble with two tables getting corrupted. One table stores my syslog entries from Kiwi Syslod Daemon, and the other stores the actual firewall logs extracted from the syslog entries. Every five minutes I run a process which reads in all the syslog entries, extracts the firewall logs from them, then deletes the entries from the syslog table. My problem is I keep getting corrupted data files, which brings the whole system to a halt. I'm running MySQL v3 (latest build) on a WinNT4 box. I'm using Perl MyODBC to process the data. Any thoughts? -Chris - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
PERROR on Windows
Is there a command like perror for the Windows version of MySQL? I can't seem to find anything to list error code descriptions in either MySQL 3 or 4. TIA - Chris - Before posting, please check: http://www.mysql.com/manual.php (the manual) http://lists.mysql.com/ (the list archive) To request this thread, e-mail [EMAIL PROTECTED] To unsubscribe, e-mail [EMAIL PROTECTED] Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
