Hi,
The code in my function below (insert_internet_usage) has suddenly dramatically slowed down and I don't understand why. The logs I'm parsing are the same size as before, but what used to take 15 minutes has shot up to about 4 hours. It's almost as if the indexes had stopped working.
I wonder if someone here can help me out please? I've tried to include all the information which I thought might be relevant below
but if you need more let me know. There haven't been any errors reported in the log since I restarted mysql. Here's my script log:
Thu Jul 1 08:56:18 2004: PARSING /sawmill/rawlog/SG_CSGL02_main_470612210000.log.gz ...
Thu Jul 1 09:00:37 2004: BULK_TABLE_INSERT ...
Thu Jul 1 09:01:43 2004: INSERT_URL_SERVERS ...
Thu Jul 1 09:02:26 2004: INSERT_URL_PATHS ...
Thu Jul 1 09:03:54 2004: INSERT_URL_QUERIES ...
Thu Jul 1 09:04:32 2004: INSERT_USER_AGENTS ...
Thu Jul 1 09:05:09 2004: INSERT_URL_VISITS ...
Thu Jul 1 09:10:06 2004: INSERT_INTERNET_USAGE ...
Thu Jul 1 09:19:31 2004: DELETE_BULK_TABLE ...
Thu Jul 1 09:19:32 2004: PARSING /sawmill/rawlog/SG_CSGL02_main_470611210000.log.gz ...
Thu Jul 1 09:29:02 2004: BULK_TABLE_INSERT ...
Thu Jul 1 09:31:36 2004: INSERT_URL_SERVERS ...
Thu Jul 1 09:33:21 2004: INSERT_URL_PATHS ...
Thu Jul 1 09:36:26 2004: INSERT_URL_QUERIES ...
Thu Jul 1 09:38:07 2004: INSERT_USER_AGENTS ...
Thu Jul 1 09:39:37 2004: INSERT_URL_VISITS ... <=== This is where the time difference has
Thu Jul 1 09:54:19 2004: INSERT_INTERNET_USAGE ... <=== suddenly increased
Thu Jul 1 13:28:37 2004: DELETE_BULK_TABLE ...
Thu Jul 1 13:28:38 2004: PARSING /sawmill/rawlog/SG_CSGL02_main_470610210000.log.gz ...
Thu Jul 1 13:39:09 2004: BULK_TABLE_INSERT ...
Thu Jul 1 13:41:59 2004: INSERT_URL_SERVERS ...
Thu Jul 1 13:43:52 2004: INSERT_URL_PATHS ...
Thu Jul 1 13:47:11 2004: INSERT_URL_QUERIES ...
Thu Jul 1 13:48:55 2004: INSERT_USER_AGENTS ...
Thu Jul 1 13:50:35 2004: INSERT_URL_VISITS ...
Thu Jul 1 14:06:55 2004: INSERT_INTERNET_USAGE ...
time now is 16:13 and it's still going.
This is the function:
sub insert_internet_usage(){
my $sql = qq {INSERT internet_usage (uid,time,ip,urlid,timetaken, cs_size,sc_size,method_ID,action_ID,virus_ID,userag
ent_ID)
SELECT bt.user, bt.time, bt.ip, uv.urlid, bt.timetaken, bt.cs_size, bt.sc_size, bt.method_ID, bt.action_ID, b
t.virus, ua.ID
FROM bulk_table bt
INNER JOIN url_servers us ON us.server=bt.server
INNER JOIN url_paths up ON up.path=bt.path
INNER JOIN url_queries uq ON uq.query=bt.query
INNER JOIN user_agents ua ON ua.useragent=bt.useragent
INNER JOIN url_visit uv ON uv.url_server_ID=us.ID
AND uv.url_path_ID=up.ID
AND uv.url_scheme_ID=bt.scheme_ID
AND uv.url_query_ID=uq.ID };
my $timenow=localtime();
print PL "\n$timenow: INSERT_INTERNET_USAGE ...";
$dbh2->do($sql) or die $dbh2->errstr;
$dbh2->commit() or die $dbh2->errstr;
}Here's a bit of extra info:
mysql> desc url_visit; +-----------------+---------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-----------------+---------+------+-----+---------+----------------+ | urlid | int(11) | | PRI | NULL | auto_increment | | url_scheme_ID | int(11) | | | 0 | | | url_server_ID | int(11) | | MUL | 0 | | | url_path_ID | int(11) | | | 0 | | | url_query_ID | int(11) | | | 0 | | | url_category_ID | int(11) | | | 0 | | +-----------------+---------+------+-----+---------+----------------+ 6 rows in set (0.01 sec)
mysql> desc internet_usage; +--------------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-------------+------+-----+---------+-------+ | uid | varchar(10) | YES | MUL | NULL | | | time | datetime | YES | | NULL | | | ip | int(10) | YES | | 0 | | | urlid | int(11) | | | 0 | | | timetaken | int(11) | YES | | 0 | | | cs_size | int(11) | YES | | 0 | | | sc_size | int(11) | YES | | 0 | | | method_ID | int(11) | YES | | 0 | | | action_ID | int(11) | YES | | 0 | | | virus_ID | int(11) | YES | | 0 | | | useragent_ID | int(11) | YES | | 0 | | +--------------+-------------+------+-----+---------+-------+ 11 rows in set (0.00 sec)
mysql> show status; +--------------------------+------------+ | Variable_name | Value | +--------------------------+------------+ | Aborted_clients | 0 | | Aborted_connects | 0 | | Bytes_received | 16714 | | Bytes_sent | 4250908493 | | Com_admin_commands | 0 | | Com_alter_table | 0 | | Com_analyze | 0 | | Com_backup_table | 0 | | Com_begin | 0 | | Com_change_db | 1 | | Com_change_master | 0 | | Com_check | 0 | | Com_commit | 2 | | Com_create_db | 0 | | Com_create_function | 0 | | Com_create_index | 0 | | Com_create_table | 0 | | Com_delete | 2 | | Com_delete_multi | 0 | | Com_drop_db | 0 | | Com_drop_function | 0 | | Com_drop_index | 0 | | Com_drop_table | 0 | | Com_flush | 0 | | Com_grant | 0 | | Com_ha_close | 0 | | Com_ha_open | 0 | | Com_ha_read | 0 | | Com_insert | 98 | | Com_insert_select | 18 | | Com_kill | 0 | | Com_load | 3 | | Com_load_master_data | 0 | | Com_load_master_table | 0 | | Com_lock_tables | 1 | | Com_optimize | 0 | | Com_purge | 0 | | Com_rename_table | 0 | | Com_repair | 0 | | Com_replace | 0 | | Com_replace_select | 0 | | Com_reset | 0 | | Com_restore_table | 0 | | Com_revoke | 0 | | Com_rollback | 0 | | Com_savepoint | 0 | | Com_select | 18 | | Com_set_option | 13 | | Com_show_binlog_events | 0 | | Com_show_binlogs | 0 | | Com_show_create | 12 | | Com_show_databases | 0 | | Com_show_fields | 14 | | Com_show_grants | 0 | | Com_show_keys | 2 | | Com_show_logs | 0 | | Com_show_master_status | 0 | | Com_show_new_master | 0 | | Com_show_open_tables | 0 | | Com_show_processlist | 0 | | Com_show_slave_hosts | 0 | | Com_show_slave_status | 0 | | Com_show_status | 2 | | Com_show_innodb_status | 0 | | Com_show_tables | 14 | | Com_show_variables | 0 | | Com_slave_start | 0 | | Com_slave_stop | 0 | | Com_truncate | 0 | | Com_unlock_tables | 1 | | Com_update | 0 | | Connections | 9 | | Created_tmp_disk_tables | 15 | | Created_tmp_tables | 16 | | Created_tmp_files | 0 | | Delayed_insert_threads | 0 | | Delayed_writes | 0 | | Delayed_errors | 0 | | Flush_commands | 1 | | Handler_commit | 0 | | Handler_delete | 0 | | Handler_read_first | 3 | | Handler_read_key | 28924102 | | Handler_read_next | 23389 | | Handler_read_prev | 0 | | Handler_read_rnd | 4649 | | Handler_read_rnd_next | 118372628 | | Handler_rollback | 0 | | Handler_update | 0 | | Handler_write | 57423139 | | Key_blocks_used | 157632 | | Key_read_requests | 2897986745 | | Key_reads | 480921 | | Key_write_requests | 13481590 | | Key_writes | 389143 | | Max_used_connections | 2 | | Not_flushed_key_blocks | 17697 | | Not_flushed_delayed_rows | 0 | | Open_tables | 19 | | Open_files | 31 | | Open_streams | 0 | | Opened_tables | 25 | | Questions | 207 | | Qcache_queries_in_cache | 2 | | Qcache_inserts | 5 | | Qcache_hits | 0 | | Qcache_lowmem_prunes | 0 | | Qcache_not_cached | 12 | | Qcache_free_memory | 33503944 | | Qcache_free_blocks | 1 | | Qcache_total_blocks | 7 | | Rpl_status | NULL | | Select_full_join | 8 | | Select_full_range_join | 0 | | Select_range | 0 | | Select_range_check | 0 | | Select_scan | 29 | | Slave_open_temp_tables | 0 | | Slave_running | OFF | | Slow_launch_threads | 0 | | Slow_queries | 23 | | Sort_merge_passes | 0 | | Sort_range | 0 | | Sort_rows | 4649 | | Sort_scan | 1 | | Table_locks_immediate | 199 | | Table_locks_waited | 0 | | Threads_cached | 0 | | Threads_created | 3 | | Threads_connected | 3 | | Threads_running | 2 | | Uptime | 28728 | +--------------------------+------------+ 132 rows in set (0.00 sec)
mysql> show index from internet_usage;
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| internet_usage | 1 | uid | 1 | uid | A | 57073 | NULL | NULL | YES | BTREE | |
+----------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
1 row in set (0.01 sec)
mysql> show index from url_visit;
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| url_visit | 0 | PRIMARY | 1 | urlid | A | 6864184 | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 1 | url_server_ID | A | NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 2 | url_path_ID | A | NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 3 | url_query_ID | A | NULL | NULL | NULL | | BTREE | |
| url_visit | 0 | url_server_ID | 4 | url_scheme_ID | A | 6864184 | NULL | NULL | | BTREE | |
+-----------+------------+---------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
5 rows in set (0.00 sec)
Thanks for any help,
js.
_________________________________________________________________
It's fast, it's easy and it's free. Get MSN Messenger today! http://www.msn.co.uk/messenger
-- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
