Re: Replaying the mysqld.log file from production onto QA???
On Thu, Sep 9, 2010 at 4:12 PM, Nunzio Daveri nunziodav...@yahoo.comwrote: So.. I am trying to mimic replaying production like queries so joins, temp tables etc... are stuff I am trying to test as well. Just doing a dump and import is no more than export and importing, I also want to test selects, updates :-) Thanks for replying :-) Then you'll have to first activate full logging on your production server. This may take quite some diskpace, and IOs, so keep the logfiles on separate spindles. Extracting the queries from the log shouldn't be all that hard; but there's too little timing information in there to do a time-true replay - and I don't know of any app that does that, either. Nunzio From: andrew.2.mo...@nokia.com andrew.2.mo...@nokia.com To: nunziodav...@yahoo.com; mysql@lists.mysql.com Sent: Wed, September 8, 2010 4:19:19 PM Subject: Re: Replaying the mysqld.log file from production onto QA??? Try using a mysqldump. - Reply message - From: ext Nunzio Daveri nunziodav...@yahoo.com Date: Wed, Sep 8, 2010 19:25 Subject: Replaying the mysqld.log file from production onto QA??? To: mysql@lists.mysql.com mysql@lists.mysql.com Hello all, I am trying to REPLAY the mysqld.log file from a production 5.1.4 server onto a QA server with 5.5 running and was wondering if there is an easy way of replaying the mysqld.log file? Just want to make sure all of the kinds of inserts, updates, selects and deletes work just as well on the 5.5 box, esp since we are turning on replication and copying tables from another server onto this server. Any ideas please? TIA... Nunzio -- Bier met grenadyn Is als mosterd by den wyn Sy die't drinkt, is eene kwezel Hy die't drinkt, is ras een ezel
Re: Replaying the mysqld.log file from production onto QA???
Hello Johan, On 9/10/2010 7:47 AM, Johan De Meersman wrote: On Thu, Sep 9, 2010 at 4:12 PM, Nunzio Daveri nunziodav...@yahoo.comwrote: So.. I am trying to mimic replaying production like queries so joins, temp tables etc... are stuff I am trying to test as well. Just doing a dump and import is no more than export and importing, I also want to test selects, updates :-) Thanks for replying :-) Then you'll have to first activate full logging on your production server. This may take quite some diskpace, and IOs, so keep the logfiles on separate spindles. Extracting the queries from the log shouldn't be all that hard; but there's too little timing information in there to do a time-true replay - and I don't know of any app that does that, either. He already did! Those are the logs he needs to replay. He has the logs already but needs tools to extract the commands and repeat them as a load test. Do you have any techniques you can share? -- Shawn Green MySQL Principal Technical Support Engineer Oracle USA, Inc. Office: Blountville, TN -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: Replaying the mysqld.log file from production onto QA???
On Fri, Sep 10, 2010 at 1:56 PM, Shawn Green (MySQL) shawn.l.gr...@oracle.com wrote: He already did! Those are the logs he needs to replay. He has the logs already but needs tools to extract the commands and repeat them as a load test. Do you have any techniques you can share? Alas, no. I suspect the rather expensive Avalanches we got might be able to, but that's not particularly helpful to him :-) I wrote a little thing many years ago that pulled Apache logs from production over SSH and multithreaded them at the staging server, but SQL stuff would be much more time-critical - can't select before the insert :-) -- Bier met grenadyn Is als mosterd by den wyn Sy die't drinkt, is eene kwezel Hy die't drinkt, is ras een ezel
RE: Replaying the mysqld.log file from production onto QA???
If possible, use web server logging to get the timings. As I explained in my previous essay, this is not at all simple or easy. If you run a benchmark more than once, be sure to reboot the system between runs. Regards, Jerry Schwartz Global Information Incorporated 195 Farmington Ave. Farmington, CT 06032 860.674.8796 / FAX: 860.674.8341 E-mail: je...@gii.co.jp Web site: www.the-infoshop.com -Original Message- From: Shawn Green (MySQL) [mailto:shawn.l.gr...@oracle.com] Sent: Friday, September 10, 2010 7:56 AM To: Johan De Meersman Cc: Nunzio Daveri; andrew.2.mo...@nokia.com; mysql@lists.mysql.com Subject: Re: Replaying the mysqld.log file from production onto QA??? Hello Johan, On 9/10/2010 7:47 AM, Johan De Meersman wrote: On Thu, Sep 9, 2010 at 4:12 PM, Nunzio Daveri nunziodav...@yahoo.comwrote: So.. I am trying to mimic replaying production like queries so joins, temp tables etc... are stuff I am trying to test as well. Just doing a dump and import is no more than export and importing, I also want to test selects, updates :-) Thanks for replying :-) Then you'll have to first activate full logging on your production server. This may take quite some diskpace, and IOs, so keep the logfiles on separate spindles. Extracting the queries from the log shouldn't be all that hard; but there's too little timing information in there to do a time-true replay - and I don't know of any app that does that, either. He already did! Those are the logs he needs to replay. He has the logs already but needs tools to extract the commands and repeat them as a load test. Do you have any techniques you can share? -- Shawn Green MySQL Principal Technical Support Engineer Oracle USA, Inc. Office: Blountville, TN -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=je...@gii.co.jp -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: Replaying the mysqld.log file from production onto QA???
Would something like: http://www.maatkit.org/doc/mk-log-player.html http://www.maatkit.org/doc/mk-log-player.htmlWork? I know the timings won't be the same as how the queries actually came in, but it could be a start, JW On Fri, Sep 10, 2010 at 8:38 AM, Johan De Meersman vegiv...@tuxera.bewrote: On Fri, Sep 10, 2010 at 1:56 PM, Shawn Green (MySQL) shawn.l.gr...@oracle.com wrote: He already did! Those are the logs he needs to replay. He has the logs already but needs tools to extract the commands and repeat them as a load test. Do you have any techniques you can share? Alas, no. I suspect the rather expensive Avalanches we got might be able to, but that's not particularly helpful to him :-) I wrote a little thing many years ago that pulled Apache logs from production over SSH and multithreaded them at the staging server, but SQL stuff would be much more time-critical - can't select before the insert :-) -- Bier met grenadyn Is als mosterd by den wyn Sy die't drinkt, is eene kwezel Hy die't drinkt, is ras een ezel -- - Johnny Withers 601.209.4985 joh...@pixelated.net
Re: Replaying the mysqld.log file from production onto QA???
The mysqld.log isn't in an executable format, but mk-query-digest can understand it and convert it into a slow query log format. Then you can use mk-upgrade to validate that your queries produce the same result. Good for you for thinking to do this -- most people just upgrade and then panic when something doesn't work right. On Wed, Sep 8, 2010 at 2:22 PM, Nunzio Daveri nunziodav...@yahoo.com wrote: Hello all, I am trying to REPLAY the mysqld.log file from a production 5.1.4 server onto a QA server with 5.5 running and was wondering if there is an easy way of replaying the mysqld.log file? Just want to make sure all of the kinds of inserts, updates, selects and deletes work just as well on the 5.5 box, esp since we are turning on replication and copying tables from another server onto this server. Any ideas please? TIA... Nunzio -- Baron Schwartz Percona Inc http://www.percona.com/ Consulting, Training, Support Services for MySQL -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: Replaying the mysqld.log file from production onto QA???
So.. I am trying to mimic replaying production like queries so joins, temp tables etc... are stuff I am trying to test as well. Just doing a dump and import is no more than export and importing, I also want to test selects, updates :-) Thanks for replying :-) Nunzio From: andrew.2.mo...@nokia.com andrew.2.mo...@nokia.com To: nunziodav...@yahoo.com; mysql@lists.mysql.com Sent: Wed, September 8, 2010 4:19:19 PM Subject: Re: Replaying the mysqld.log file from production onto QA??? Try using a mysqldump. - Reply message - From: ext Nunzio Daveri nunziodav...@yahoo.com Date: Wed, Sep 8, 2010 19:25 Subject: Replaying the mysqld.log file from production onto QA??? To: mysql@lists.mysql.com mysql@lists.mysql.com Hello all, I am trying to REPLAY the mysqld.log file from a production 5.1.4 server onto a QA server with 5.5 running and was wondering if there is an easy way of replaying the mysqld.log file? Just want to make sure all of the kinds of inserts, updates, selects and deletes work just as well on the 5.5 box, esp since we are turning on replication and copying tables from another server onto this server. Any ideas please? TIA... Nunzio
Replaying the mysqld.log file from production onto QA???
Hello all, I am trying to REPLAY the mysqld.log file from a production 5.1.4 server onto a QA server with 5.5 running and was wondering if there is an easy way of replaying the mysqld.log file? Just want to make sure all of the kinds of inserts, updates, selects and deletes work just as well on the 5.5 box, esp since we are turning on replication and copying tables from another server onto this server. Any ideas please? TIA... Nunzio
Re: Replaying the mysqld.log file from production onto QA???
Try using a mysqldump. - Reply message - From: ext Nunzio Daveri nunziodav...@yahoo.com Date: Wed, Sep 8, 2010 19:25 Subject: Replaying the mysqld.log file from production onto QA??? To: mysql@lists.mysql.com mysql@lists.mysql.com Hello all, I am trying to REPLAY the mysqld.log file from a production 5.1.4 server onto a QA server with 5.5 running and was wondering if there is an easy way of replaying the mysqld.log file? Just want to make sure all of the kinds of inserts, updates, selects and deletes work just as well on the 5.5 box, esp since we are turning on replication and copying tables from another server onto this server. Any ideas please? TIA... Nunzio
