[Wikitech-l] [Selenium] Issue with importing a test database
Hello everybody, for the Selenium Framework I have a very specific database related issue which is hard for me to decide. This is the problem: In order to have a fresh state for every test, we agreed to have a test database (and image folder, but this is a sidetrack now) for every test suite run. The fresh database is created from a SQL file which can be attached to a test as a resource. Now, to make the creation of such SQL files as easy as possible, I wanted to be able to simply use SQL dumps created with mysqldump. The import of the data is done via the existing databse abstraction layer. I use the method DatabaseBase::sourceFile which in turn calls DatabaseBase::sourceStream. The problem is now, that some of the SQL INSERT statements seem to be too long for this method. Platonides pointed me to the source of the problem (thanks!). It lies currently in line 2506 (Database.php) : $line = trim( fgets( $fp, 1024 ) ); So the lines read are limited to 1024 characters. If I remove this limitation, everything works fine. PHP manual tells me that the length parameter is optional as of PHP version 4.2.0. Since I don't know enough about how fgets works and what its security issues are, I wonder, is there a reason not to remove the parameter? Cheers, Markus (mglaser) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Selenium] Issue with importing a test database
Markus Glaser wrote: Hello everybody, for the Selenium Framework I have a very specific database related issue which is hard for me to decide. This is the problem: In order to have a fresh state for every test, we agreed to have a test database (and image folder, but this is a sidetrack now) for every test suite run. The fresh database is created from a SQL file which can be attached to a test as a resource. Now, to make the creation of such SQL files as easy as possible, I wanted to be able to simply use SQL dumps created with mysqldump. The import of the data is done via the existing databse abstraction layer. I use the method DatabaseBase::sourceFile which in turn calls DatabaseBase::sourceStream. The problem is now, that some of the SQL INSERT statements seem to be too long for this method. Platonides pointed me to the source of the problem (thanks!). It lies currently in line 2506 (Database.php) : $line = trim( fgets( $fp, 1024 ) ); So the lines read are limited to 1024 characters. If I remove this limitation, everything works fine. PHP manual tells me that the length parameter is optional as of PHP version 4.2.0. Since I don't know enough about how fgets works and what its security issues are, I wonder, is there a reason not to remove the parameter? Cheers, Markus (mglaser) I think it can be removed safely. Although in this case I would just run mysqldump with --skip-extended-insert so that it doesn't create such long lines. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Selenium] Issue with importing a test database
Hi, I think it can be removed safely. Great to hear! Although in this case I would just run mysqldump with --skip-extended-insert so that it doesn't create such long lines. Yes, I tried that. But there are tables like l10ncache or objectcache that store serialized objects which produce long lines even in that case. Still, I think that should be a recommendation for creating the dumps. Cheers, Markus ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Selenium] Issue with importing a test database
Although in this case I would just run mysqldump with --skip-extended-insert so that it doesn't create such long lines. Yes, I tried that. But there are tables like l10ncache or objectcache that store serialized objects which produce long lines even in that case. Still, I think that should be a recommendation for creating the dumps. Do you really want to dump those caching tables? Good question... I don't think it's generally necessary. But then again, I think there are two reasons why we should allow for dumps that contain these tables: * it's easy to use plain old mysql dumps. The easier we make it for developers to test their code, the better. An alternative would be to provide a dumper script that removes these tables or doesn't dump them in the first place. I could dive into that if people here think that's a better way to go. * maybe someone wants to write a regression test that reproduces some specific caching issue. I agree this is unlikely, most probably unit tests would be a better way here. But then again, the mechanism I am working on might be a possible basis for unit test resources as well. The point is (and I did not see that in my previous post) that even if I skip caching tables in the import, there is most certainly one other table which might exceed the 1024 byte limit and that is text. Cheers, Markus ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
