Re: subqueries *not* using indexes for IN clause
Greg Whalin wrote: We have noticed this as well and it is really pretty shoddy. It seems that when using IN( SELECT ), they treat it as ANY() which does a full table scan. Only way we have found to get fast performance out of subqueries is to use the derived table format and join with the derived table. But if I have to do that, might as well just use the join without the funky syntax. Still, it does simplify some sql which is difficult to do with a regular join (i.e. joining w/ max() col, etc.). In any rate, I agree. What is the point of claiming to offer sub-selects when thay are practically unusable in IN() statements which is how most people use subselects IMO. Yup... couldn't agree more! MySQL subqueries in 4.1 are at best useless and at worst Evil.. plain Evil ! ;) But nice try guys! This seems like it REALLY deserves a bug fix! Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
subqueries *not* using indexes for IN clause
http://www.peerfear.org/rss/permalink/2005/04/02/BrokenMySQLSubqueries Whats up with this? As far as I can tell MySQL subqueries in 4.1.x releases are totally broken with IN clauses The major reason is that they don't use *ANY* indexes and resort to full table scans. Lets take two queries: mysql> EXPLAIN SELECT * FROM FEED, ARTICLE WHERE ARTICLE.ID = 1628011 AND FEED.ID = ARTICLE.ID *** 1. row *** id: 1 select_type: SIMPLE table: FEED type: const possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 1 Extra: *** 2. row *** id: 1 select_type: SIMPLE table: ARTICLE type: const possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 1 Extra: 2 rows in set (0.00 sec) Which is *great*. The join is using both of the PRIMARY indexes on the columns and only references one row. Can't get any better than that! Now lets rewrite the SELECT to use a subquery: mysql> EXPLAIN SELECT * FROM FEED WHERE ID IN (SELECT ID FROM ARTICLE WHERE ID = 1628011) *** 1. row *** id: 1 select_type: PRIMARY table: FEED type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 2316698 Extra: Using where *** 2. row *** id: 2 select_type: DEPENDENT SUBQUERY table: ARTICLE type: const possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 1 Extra: Using index 2 rows in set (0.00 sec) And here's where the fun begins. The FEED table won't use *ANY* index! It really can't get ANY worse than that. So either this is a bug in both 4.1.10 and 4.1.7 or the optimizer is just plain broken. Note that using FORCE INDEX doesn't work at all. -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
Transparent schema update for replicated installs
(Did I send this off to the list already? It doesn't seem to be updated! ) I've been thinking recently about supporting transparent schema update of production systems in order to deliver zero downtime. What I'm thinking of is a scenario where you use a load balancer to take one of the slaves, alter its schema, and then promote it to master. It should be possible to perform runtime updates of schema with the following steps: * Starting with a cluster of MySQL boxes replicated to a master as slaves. * Take one of the MySQL boxes which is curently setup to replicate from the master and run all ALTER TABLE commands on it. * Then let it catch up to the master so that its only a few seconds behind. * Then run the ALTER TABLE commands on the other MySQL slaves one at a time (or concurrently) (each after they've been taken out of production). * Promote the current slave to the current master by doing all the normal config of a master and then disabling the slave setup. * Change the load balancer to have the NEW master be the ALTER'd slave From: masterdb.server.com -> 10.0.0.2 To: masterdb.server.com -> 10.0.0.3 * This will allow new MySQL clients to start using the new master (which is now ready but has locks blocked). * Then lock the tables on the current master. * Kill ALL connections of the current master so that the slaves and clients reconnect. There are a few problems with this approach (but it shows some serious potential) * What if your clients are in the middle of a transaction? They'll have to failover correctly to the new master. * This really isn't LIVE alter as it would take a few seconds for everything to switch over. * The slaves will need their binary log settings fixed (or can I just run RESET MASTER) to correct these? * What other steps am I missing? -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Automatic server-id generation for slaves?
Right now one of the only reasons we can't put our entire config for our slaves in CVSup is that the config *requires* the ability to set a server-id for each machine. Seems like it would be pretty trivial to support a hostname based policy for this. You could simply look at the IP/hostname and set the value from this (though you might need a tracking table). Policies could include: - IP based server-id (IPs are 32bit) - parse the hostname for an ID (db4.server.com would yield a server-id of 4) - Adler32/SHA1 truncate the hashcode of the hostname The first two seem sufficient. This wouldn't be the default of course and would require an explicit config. Thoughts? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Repair with keycache
Mohamed Badri wrote: thank you for the links, I can't change system variables at runtime, so the only choice I have is, probably, to stop mysql server set variables and then run another ALTER TABLE. ;-( I'm going to siwtch to mysql4 as soon as possible. OH!... yeah... if you're on a <4 ver of mysql then I don't know what to tell you ;) Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Repair with keycache
Mohamed Badri wrote: Hi, just had a problem with a myisam table who reached 4GB of data, I increased the number of rows by doing : ALTER TABLE foo MAX_ROWS=10 This is EXACTLY what you want: http://www.peerfear.org/rss/permalink/2004/10/16/MySQLAndALTERTABLEGuiltyAsCharged http://www.peerfear.org/rss/permalink/2004/10/15/MaxTableSizeInMySQL REPAIR TABLE considered harmful would be a better blog post title for these guys! Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: bash powered MySQL Queries
Edward Macnaghten wrote: Funny you should mention that May I guide you to my recent announcment of ShellSQL 0.7 - web page at http://www.edlsystems.com/shellsql - a utility to do just as you want (I think) released under GPL. It must be good - I wrote it myself :-) SELECT * FROM /dev/zero INTO OUTFILE /dev/null :) Seems like a cool project though. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Connection latency reduction attempt for load balancing mysql slaves.
I'm experimenting with using ConnectorJ 3.1.7 to connect to MySQL slaves to handle load balancing queries. The environment would either include a hardware loadbalancer or round robin DNS (or static configuration). The main component though is having the JDBC driver continually reconnect (to actually balance the load) The major problem is that we run a LOT of small queries to generate our pages so per-query latency is very important. After benchmarking continual reconnect to my laptop running MySQL 4.1.9 and with our DB connection pool I'm noticing a 14x performance gap (with manual reconnects still loosing). Here are the results: // Total parse count: 1000 // Total duration: 3173 milliseconds // Total avg duration: 3.173 milliseconds // Total per second: 315.15915 // Testing method: test2 // // Total parse count: 1000 // Total duration: 217 milliseconds // Total avg duration: 0.217 milliseconds // Total per second: 4608.295 So with our DB pool each query only takes .217 ms but without it its taking 3.173 ms. This doesn't sound like much but if you multiply that by 1000x queries its significant and can seriously screw with total throughput. I've also tried removing any queries and JUST connection (without a SELECT 1) test and it still takes 2.8ms. Here's my JDBC connection string: jdbc:mysql://127.0.0.1/mysql?useUnicode=true&characterEncoding=UTF-8&useOldUTF8Behavior=true&holdResultsOpenOverStatementClose=true&cacheServerConfiguration=true&elideSetAutoCommits=true I've also increase thread_cache_size on the server and Thread_created is a static value. This only had a slight performance advantage. Any ideas? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: MySQL Load Balancing w/ Alteons...Half Open Connections
Ian Sales (DBA) wrote: Kevin A. Burton wrote: Define DoS? - Denial of Service... ug... Thats not what I meant... I mean what type of behavior were you noticing? Just all connections being occupied on the server? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: MySQL Load Balancing w/ Alteons...Half Open Connections
Jason J. W. Williams wrote: Has anyone ever had a problem with Alteon load balancers leaving the MySQL connections half open? After about a minute of heavy use the Alteon has completely DoS'd our MySQL servers. I know we must be doing something wrong...just not sure what. Any help is greatly appreciated! Define DoS? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Using a Load balancer with MySQL 4.1.7 and JDBC 3.1.7 (performance)...
I'm playing around with using a load balancer to balance our slaves. What I want to do is put a load balancer in front of our slaves and then reconnect each time via the JDBC driver. This way a slave can fail and I can just take it out of service. It also means that I can keep performance up at a decent level. The problem is that my benchmarks show thats its MUCH slower to use this even without the load balancer. With the JDBC driver connecting each time and running SELECT 1 it takes 7150ms. When using our connection pull it only takes 200ms (this is with 100 queries). I increased thread_cached_size to 80 on the server and I notice that when I do a SHOW STATUS that Threads_created isn't increasing so thats not a problem. So that leaves me with the following questions: 1. Are there any other variables or settings I should tweak to make this faster? 2. Is this as fast as its going to get? 3. Does actually USING the loadbalancer hold the connection open thus speed it up? If it does hold the connection open how does the JDBC driver handle this situation? Seems like it would get confused. 3. Is there a HOWTO for using MySQL with a load balancer? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: allowMultQueries for Connector-J breaks replication from master.
Mark Matthews wrote: <>Kevin, 4.0.12 doesn't have the feature, so you should've gotten a 'syntax error' when you tried it ;) Hm.. Maybe you're right... Now that you mention it I did test it against 4.1.7. -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
Re: allowMultQueries for Connector-J breaks replication from master.
Kevin A. Burton wrote: Not sure which list this should go to ... seems like a bad bug so I sent it to the java and mysql lists to prevent it from hitting anyone else. (I also searched bugs.mysql.com which doesn't seem to show anything). Actually... one more note The last time I tested this was with MySQL 4.0.12... it worked fine. This bug was only raised in MySQL 4.1.7 Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
allowMultQueries for Connector-J breaks replication from master.
Not sure which list this should go to ... seems like a bad bug so I sent it to the java and mysql lists to prevent it from hitting anyone else. (I also searched bugs.mysql.com which doesn't seem to show anything). We're playing with the allowMultiQueries feature in Connector-J 3.1.7 which allows you to run multiple queries in one executeStatement... this can reduce the time for 1000 queries from 1000ms down to about 1ms which for batch updates it MUCH faster. (its also faster than PreparedStatement batch updates by about 1000x as each stmt in the batch takes 1ms). In my tests I ran these against a single master but it looks like these will break replication. One of our engineers was playing with using this on our master and the query immediately broke all of our slaves: Last_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '; UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 1061 WHERE USER_FEED.USE' at line 1' on query. Default database: 'ksa'. Query: 'UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 5239 WHERE USER_FEED.USER_ID = 79 AND USER_FEED.FEED_ID = 61025 AND USER_FEED.SUBJECT_ID >= 0 ; UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 1061 WHERE USER_FEED.USER_ID = 79 AND USER_FEED.FEED_ID = 61026 AND USER_FEED.SUBJECT_ID >= 0 ; UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 4255 WHERE USER_FEED.USER_ID = 79 AND USER_FEED.FEED_ID = 61027 AND USER_FEED.SUBJECT_ID >= 0 ; UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 400 WHERE USER_FEED.USER_ID = 79 AND USER_FEED.FEED_ID = 61028 AND USER_FEED.SUBJECT_ID >= 0 ; UPDATE USER_FEED SET USER_FEED.UNREAD_ARTICLE_COUNT = 1007 WHERE USER_FEED.USER_ID = 79 AND USER_FEED.FEED_ID = 61261 AND USER_FEED.SUBJECT_ID >= Skip_Counter: 0 He ran the statement once with about 200 UPDATEs in the multiquery. It seems like the master wrote these as 200 FULL statements (not split into individual statements) so when the slave replayed the transaction it broke. The issue seems to be that while the SQL is executed correctly its not written to the binary log correctly and thus breaks all the masters. Any thoughts here? Obviously we can't move to allowMultiQueries since this seems like a fatal bug. Thanks! Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Memory tables much slower in 4.1.7 ?
Dathan Pattishall wrote: Hmm that's a range, that should do a table scan in 4.0.18, Yes... I believe it did but since its a memory table it went by really quick. since a memory table type is just a hash table. In 4.1 I believe it supports ranges since the table is more of a myISAM type. Yes... but only if an index is used... we don't have a btree index here... Is there an index on TIMESTAMP? No... Not a btree index. There are hash indexes but on other cols so they won't be used. Does the range cover more then 30% of the table? Yes... it does a full table scan. I rewrote it as an EXPLAIN: mysql> EXPLAIN -> SELECT * FROM FOO_MEMORY WHERE FOO_MEMORY.TIMESTAMP < 1105055409729; ++-+-+--+---+--+-+--+-+-+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows| Extra | ++-+-+--+---+--+-+--+-+-+ | 1 | SIMPLE | FOO_MEMORY | ALL | NULL | NULL |NULL | NULL | 3505527 | Using where | ++-+-+--+---+--+-+--+-+-+ 1 row in set (0.00 sec) 3505527 is the full size of our table so.. its doing a full table scan. Very strange... Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Memory tables much slower in 4.1.7 ?
Under 4.0.18 we were loading about 800M of data into a memory table to get better performance from some of our main queries. Via crontab we would DELETE older links in the memory table and then INSERT links from a myisam table. This process under 4.1.7 is MUCH slower. Specifically the DELETE is taking forever. We would run: DELETE FROM FOO_MEMORY WHERE FOO_MEMORY.TIMESTAMP < 1105055409729; But now this takes forever... I realize that no index is used on this query but due to the fact that its a memory table it should sitll be FAST as heck. Hash anything changed with rehashing that could effect the performance here? I could install a 4.0.18 node to benchmark but this would take a while as they are all upgraded... Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: What is the difference between 4.1 and MAXDB 7.5?
Logan, David (SST - Adelaide) wrote: Hi Juan, MAXDB is a totally different product, more akin to the "corporate" databases like Oracle, Informix, Sybase etc. A description from the mysql web site ( www.mysql.com ) MaxDB(tm) by MySQL For the most demanding enterprise applications, MySQL offers MaxDB by MySQL. Formerly known as SAP DB, MaxDB is the result of a strategic alliance between MySQL and SAP to jointly develop and market an enterprise-class Open Source database capable of running high-end business-critical applications including SAP/R3. MaxDB complements the MySQL database, and is SAP-certified. It includes features such as stored procedures, triggers and views for the most demanding enterprise use. Call me crazy but doesn't having two databases just end up confusing your customers. Actually now you have 3 if you include NDB -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: MySQL 4.1.7 allows non-unique values in a unique index?!
Jocelyn Fournier wrote: Hi, This could also be a badly corrupted table, what does CHECK TABLE / REPAIR TABLE report ? Thats next on our list... going to take care of that tonight. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: MySQL 4.1.7 allows non-unique values in a unique index?!
Kevin A. Burton wrote: WOW! This is a really bad bug... So now only that... but check this out: mysql> SELECT * FROM LITERAL WHERE VALUE = 'Law'; +--+---+ | ID | VALUE | +--+---+ | 14076840 | Law | +--+---+ 1 row in set (0.00 sec) Which only returns ONE row even though there are two in the database with the same value! Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
MySQL 4.1.7 allows non-unique values in a unique index?!
WOW! This is a really bad bug... mysql> SHOW INDEX FROM LITERAL; +-++--+--+-+---+-+--++--++-+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +-++--+--+-+---+-+--++--++-+ | LITERAL | 0 | PRIMARY |1 | ID | A | 14331755 | NULL | NULL | | BTREE | | | LITERAL | 0 | VALUE|1 | VALUE | A | 14331755 | NULL | NULL | | BTREE | | +-++--+--+-+---+-+--++--++-+ Notice the unique index on VALUE ? Now take a look at this: mysql> SELECT *, MD5(LITERAL.VALUE) AS MD5_VALUE FROM LITERAL WHERE ID=567344 OR ID=14076840; +--+---+--+ | ID | VALUE | MD5_VALUE| +--+---+--+ | 567344 | Law | 81588d326cebe6416d3904db93603af1 | | 14076840 | Law | 81588d326cebe6416d3904db93603af1 | +--+---+--+ 2 rows in set (0.00 sec) Seems like a fatal bug to me! Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: use of soundex in queries
Raphael Matthias Krug wrote:
Hi,
I need to compare names from different tables and therefore I need
to know the proper use of soundex. I googled for it, but could not find
anything useful. And select soundex('text') is no help for me.
FYI Soundex has real problems ... use DoubleMetaphone...
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Master will insist on running as a SLAVE if data/master.info exists.
Mikael Fridh wrote: Kevin A. Burton wrote: This is a bug. Feature. Putting system configuration information on a unix machine in /var.. .yeah... thats not a feature. In fact, you don't need those configuration statements in the first place. You could just as well initiated the slave replication by issuing CHANGE MASTER TO... Yes... I realize Slave is always "started" unless my.cnf says "skip-slave-start". Anyway, WITH skip-slave-start you will still have the slave information (binlog positions etc.) initiated but it will just not start replicating. The information in master.info overrides anything in my.cnf. Again... and this file is in /var... Whats the point of /etc/my.cnf... why not just store everything in /var? Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Master will insist on running as a SLAVE if data/master.info exists.
This is a bug. I have all the slave configuration in /etc/my.cnf commented out. However the machine was ONCE a slave and so data/master.info exists. I commented out all the /etc/my.cnf settings WRT master-* yet when I restart the box it STILL insists on starting the slave via master.info If these settings aren't enabled the slave shouldn't be started. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Unable to shutdown slave when master isn't available.
We have a failed master right now... If I attempt to shutdown any of my slaves right now they ALL timeout when trying to shutdown. To make matters MUCH worse I can no longer connect to the STILL running mysql daemon. So the only way to restart is with a killall -9 which results in a corrupt database on the slave. This is REALLY bad! I'm sure MySQL could do a better job here. No need for a corrupt database. Why not have the control port be the LAST thing to shutdown so if it fails you can still connect and FLUSH TABLES. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
How to speed up ANALYZE TABLE?
The ANALYZE TABLE entry in the manual doesn't say anything about performance optimization. How do I get optimum performance out of ANALYZE TABLE? Can I just set the same variables as I would with REPAIR TABLE? key_buffer_size sort_buffer_size myisam_sort_buffer_size Also... Does A.T. always need to read ALL rows from the table? I assume so. Also... why does it require *any* type of lock? For large tables the chance that data would be modified during the ANALYZE to cause the results to be wildly innacurate are rare. It seems possible to just let ANALYZE run without any locks and just let the data be off by a few K rows. Which seems MUCH better than NEVER running them. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
MySQL load balancing...
Was curious what people on the list are using for load balancing.. there are a number of techniques here but it would be interesting if people could share some real-world experiences HTTP load balancing is pretty well understood but there's not a bunch out there on MySQL load balancing Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
unions will full column names won't work in 4.1.7
Not sure if this is a bug... probably should be. On 4.1.18 I can run: (SELECT * FROM FOO WHERE ID = 1) UNION (SELECT * FROM FOO WHERE ID = 2) ORDER BY FOO.COL_A Which will work just fine However when I use this query on 4.1.7 I get ERROR 1250 (42000): Table 'ARTICLE' from one of the SELECTs cannot be used in global ORDER clause I have to rewrite it to use: (SELECT * FROM FOO WHERE ID = 1) UNION (SELECT * FROM FOO WHERE ID = 2) ORDER BY COL_A .. see the change in the ORDER BY... I can't call if FOO.COL_A I have to call it COL_A Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: YOUR EMAIL ADDRESS HAS BEEN ADDED TO MY WHITE LIST
[EMAIL PROTECTED] wrote: Your email address has been added to my Spam Fighter White List. Adding your address to the White List ensures that I will always receive email you send to me. How do I unsubscribe from your Spam Figher White List spam?! :) Kevn -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Strange results
Steve Grosz wrote: If you can tell me the command to dump the table format, I'm more than happy to list it here. SHOW CREATE TABLE FOO; -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Load Index Into Cache is broken! ... and a workaround
Dan Nelson wrote: In the last episode (Dec 11), Philip Lane said: I saw a couple of posts about this and thought I'd join the chorus. Running version 4.1.7 on a WinXP system. I have a table "mindata" containing just over 1 million lines of price/volume data for exchange-traded stocks. It's for a .NET web application. There are about 3500 different stocks in the table. Each has it's own ID number, and each has about 300 lines corresponding to different dates: When I first turn on the computer queries are disturbingly slow! mysql> Select * from mindata where ID = 2345; +--++---+---+---++---+---+ | ID | TimeStamp | Close | Chg | Open | Volume | Low | High | +--++---+---+---++---+---+ | 2345 | 2003-09-15 | 17.55 | -0.1 | 17.64 | 19200 | 17.5 | 17.64 | | 2345 | 2003-09-16 | 17.58 | 0.03 | 17.5 | 9300 | 17.49 | 17.65 | | 2345 | 2003-09-17 | 17.5 | -0.08 | 17.53 | 6500 | 17.46 | 17.6 | 321 rows in set (1.58 sec) I'd say 95% of that time is spent reading the records, not reading the index. Try timing "select count(*) from mindata where ID = 2345". Just a note... your query times here can't be trusted. You need to do SELECT SQL_NO_CACHE * FROM ... Because if you don't the second time you run the query it will take 0.0ms to return it. Also loading the cache into memory won't speed up the SELECT * portion becuase it needs to read rows from disk and disk is evil. The second time you run the query the filesystem buffer cache will probably have the blocks in memory. Note sure about WinXP as this is a red flag. But Linux will certainly do the right thing. If you're still seeing performance hits try to do an OPTIMIZE TABLE to get contiguous blocks on disk. If the table is small enough (32M did you say?) just use a HEAP table and load it all in memory. This will CERTAINLY keep your index in memory and all the rows will be in memory too so you'll get constant fast times... Read up on memory tables... there are some gotchas you need to be careful of.. (like the data won't be there when you restart). Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
Re: MySQL 4.1.7 Network slowdown
Donny Simonton wrote: It was probably attempting to do a reverse and nothing exists, so it just has to timeout. Reverse DNS is evil... :) http://dev.mysql.com/doc/mysql/en/DNS.html You can disable DNS hostname lookups by starting |mysqld| with the |--skip-name-resolve| option. However, in this case, you can use only IP numbers in the MySQL grant tables. If you have a very slow DNS and many hosts, you can get more performance by either disabling DNS lookups with |--skip-name-resolve| or by increasing the |HOST_CACHE_SIZE| define (default value: 128) and recompiling |mysqld|. Another issue... why was this even having a problem? Are you not running JDBC connection pooling? This should have only happened once and then moved on since the connection is open. I don't think MySQL is doing this per query. Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
SHA1 option for Base64+filesafe.
Right now mysql's SHA1 implementation uses Base16 encoding... Why not add a Base64+filesafe encoding. This results in significant space storage. Maybe call the function SHA1BASE64() Example: Base64+filesafe(SHA1): C-7Hteo_D9vJXQ3UfzxbwnXaijM Base64(SHA1):C~7Hteo-D9vJXQ3UfzxbwnXaijM Base32(SHA1):BPXMPNPKH4H5XSK5BXKH6PC3YJ25VCRT Base16(SHA1):0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33 Base64+filesafe is just an encoding that removes / and other chars that can't be in URLs and filenames. I don't know about you but I like my database as less bloated as possible ;) Kevin -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
ALTER table performance and bugs...
I just posted two detailed issues WRT the perfomance of ALTER table and what I think are pretty significant issues. For the last few days I've been using MySQLs ALTER and REPAIR table functionality and its caused tons of countless problems and a great deal of lost sleep. The first problem I noticed was that for large tables ALTER TABLE was taking hours! Lets say you have a 30G table. Good luck altering it as the default MySQL configuration will probably take 100 or more hours. In MySQLs defense there are a number of variables you can use to increase the performance of an ALTER but the problem is that the two major ones (myisam_max_extra_sort_file_size, and myisam_max_sort_file_size) can't be set at runtime (during an ALTER). If these are set too low MySQL will revert to a /"Repair with keycache"/ strategy and this is 1000x slower than "Repair with sort". Once MySQL selects a ALTER strategy it can't use a faster one and you just have to ride it out. http://www.peerfear.org/rss/permalink/2004/10/16/MySQLAndALTERTABLEGuiltyAsCharged ... and ... It turns out that the performance of ALTER TABLE is somewhat of a black art and kept secret by the MySQL high priests <http://www.lordblink.com/> . First off it seems there are two problems: # In a replicated environment the ALTER TABLE is actually run on the master first and *then* the slave. This is not what you want because this will take twice as long. There needs to be a way to say /"don't run this on the slave"/ which would allow you to then ssh into your slave and run the ALTER TABLE directly. Of course if I could get the ALTER TABLE to run fast enough then this wouldn't be a problem. http://www.peerfear.org/rss/permalink/2004/10/15/MaxTableSizeInMySQL Interested in feedback... -- Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an invite! Also see irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html If you're interested in RSS, Weblogs, Social Networking, etc... then you should work for Rojo! If you recommend someone and we hire them you'll get a free iPod! Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
