Re: [Wikitech-l] [Labs-l] Maria DB
On 14/02/13 18:26, Faidon Liambotis wrote: > Ubuntu has experimented in the past with the concept of automatically > generating and shipping symbols for *all* packages, packaged up in a > "ddeb"s (same format as .deb) and shipped via a different repository > that isn't mirrored by all of the downstream mirrors. > > This was years ago, I'm not sure what has happened since then. I > remember being discussed in Debian as well, but it was never adopted, > probably because noone ever implemented it :) Good question. There are a few bugs and blueprints about it, and they show as *implemented* https://blueprints.launchpad.net/ubuntu/+spec/apt-get-debug-symbols https://bugs.launchpad.net/ubuntu/+bug/14484 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2006-September/000195.html What happened, then? ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
I would much rather abandon using debs than use what the debian project has done to mysql packaging in any production environment. If the discussion has come down to this, I did WMF a disservice by drifting away from Domas' optimized "make ; make install ; rsync unstripped binaries to prod" workflow. In general, I find environments that don't individually package according to distro standards every part of their core application stack that gets built in-house to be more productive, and more responsive to the needs of developers and ultimately the application. When an ops team claims that building a recent version of libmemcached for a stable OS is almost impossibly hard and will take weeks because it requires backporting a debian maintainers packaging of it for an experimental distro with that distros unrelated library version dependencies and reliance on a newer incompatible dpkg tool chain, there's probably something wrong with that workflow. I like to rely on Linux distros for the lowest common denominator layer of the stack and related security updates. The approach that goes into building and maintaining such a beast are rather different than the concerns that go into operating a continually developed and deployed distributed application used by half a billion people. I don't see a win in trying to force the two together. On Thursday, February 14, 2013, Faidon Liambotis wrote: > > > For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg > package by default. That's a shame, we can ask for that. As for the rest > of Asher's changes, I'd love to find a way to make stock packages work > in our production setup, but I'm not sure if the maintainer would > welcome the extra complexity of conditionally switching behaviors. We > can try if you're willing to, Asher :) > > Regards, > Faidon > > ___ > Labs-l mailing list > lab...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/labs-l > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
On Thu, Feb 14, 2013 at 05:14:31PM +0100, Mark Bergsma wrote: > Debug information is *highly useful* in a production setup, and we try > to run all our core applications with it so we have a chance to debug > issues when they occur. > > I think the only reason distributions omit debug information is to > save disk space. A lot of Debian packages ship -dbg alongside the main package, that contains the stripped-out debug symbols in /usr/lib/debug (gdb loads those automatically, either based on the filename or build-id). The toolchain handles this more or less automatically, but it still needs maintainer action to define this separate package and upload it with every package upload. Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a "ddeb"s (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors. This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :) For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg package by default. That's a shame, we can ask for that. As for the rest of Asher's changes, I'd love to find a way to make stock packages work in our production setup, but I'm not sure if the maintainer would welcome the extra complexity of conditionally switching behaviors. We can try if you're willing to, Asher :) Regards, Faidon ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
On Feb 14, 2013, at 5:02 PM, Petr Bena wrote: > Keeping debug symbols in binaries will result in poor performance, or it > should That's bollocks. It results in a larger binary _on disk_. The symbol table isn't even loaded into memory and doesn't affect performance. Debug information is *highly useful* in a production setup, and we try to run all our core applications with it so we have a chance to debug issues when they occur. I think the only reason distributions omit debug information is to save disk space. -- Mark Bergsma Lead Operations Architect Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
Er, no it shouldn't. Initial execution might take microseconds longer due to larger binary sizes and the elf loader having to skip over the symbols but that's about it. On Thursday, February 14, 2013, Petr Bena wrote: > Keeping debug symbols in binaries will result in poor performance, or it > should > > > On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman > > > wrote: > >> For most projects, I recommend using the official packages available via >> the MariaDB projects own apt repo. >> >> The official packages are based on the Debian mysql packaging where >> installing the server package also installs a default database created >> around generic config defaults, a debian mysql maintenance user with a >> randomly generated password, and scripts (including init) that assume >> privileged access via that user. That is, installing the packages provides >> you with a fresh running working database with generic defaults suitable >> for a small server, and certain admin tasks automated. I think that's what >> the average labs and general users wants and expects. >> >> The packages I've built for production use at wmf strips out all of the >> debianisms, the debian project script rewrites, the pre/post install >> actions. They also leave debug symbols in the binaries and have compiler >> flag tweaks, but do not at this stage contain any source >> patches. Installing the server package doesn't create a default db, or >> provide an environment where you can even start the server on a fresh >> sever >> install without further work. Probably not a good choice for most labs >> users. >> >> On Wednesday, February 13, 2013, Petr Bena wrote: >> >> > thanks for updates. >> > >> > Can you tell me what is a difference between maria db you are using and >> > the version that is recommended for use on ubuntu? >> > >> > >> > On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman >> > > > 'afeld...@wikimedia.org');>> 'cvml', 'afeld...@wikimedia.org > 'afeld...@wikimedia.org');>');> >> > > wrote: >> > >> >> The production migration to MariaDB was paused for a time by the EQIAD >> >> datacenter migration and issues involving other projects that took up >> my >> >> time, but the trial production roll-out will resume this month. All >> signs >> >> still point to our using it in production. >> >> >> >> I did a lot of query testing on an enwiki MariaDB 5.5 slave over the >> >> course >> >> of more than a month before the first production deployment. Major >> >> version >> >> migrations with mysql and derivatives are not to be taken lightly in >> >> production environments. At a minimum, one must be concerned about >> query >> >> optimizer changes making one particular query type significantly >> slower. >> >> In the case of the switch to 5.5, there are several default behavior >> >> changes over 5.1 that can break applications or change results. Hence, >> >> some serious work over a plodding time frame before that first >> production >> >> slave switch. >> >> >> >> Despite those efforts, a couple weeks after the switch, I saw a query >> >> generated by what seems to be a very rare edge case from that AFTv4 >> >> extension that violated stricter enforcement of unsigned integer types >> in >> >> 5.5, breaking replication and requiring one off rewriting and >> execution of >> >> the query locally to ensure data consistency before skipping over it. >> I >> >> opened a bug, Mathias fixed the extension, and I haven't seen any other >> >> compatibility issues from AFTv4 or anything else deployed on enwiki. >> >> >> >> That said, other projects utilize different extensions, so all of my >> >> testing that has gone into enwiki cannot be assumed to fully cover >> >> everything else. Because of that, and because I want to continue >> >> proceeding with caution for all of our projects, this will continue to >> be >> >> a >> >> slow and methodical process at this stage. Bugs in extensions that >> aren't >> >> used by English Wikipedia may be found and require fixing along the >> way. >> >> >> >> As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. >> >> >> >> Best, >> >> Asher >> >> >> >> On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena >> >> > >> 'benap...@gmail.com');>> 'cvml', 'benap...@gmail.com > 'benap...@gmail.com');>');>> >> >> wrote: >> >> >> >> > Okay - so what is outcome? Should we migrate beta cluster? Are we >> going >> >> to >> >> > use it in production? >> >> > >> >> > >> >> > On Wed, Feb 13, 2013 at 2:08 PM, Chad >> >> > > >> > 'innocentkil...@gmail.com');>> 'cvml', 'innocentkil...@gmail.com > 'innocentkil...@gmail.com');>');>> >> >> wrote: >> >> > >> >> >> On Wed, Feb 13, 2013 at 8:05 AM, bawolff >> >> >> > >> >> 'bawolff%2...@gmail.com');>> 'cvml', 'bawolff%2...@gmail.com > 'bawolff%252...@gmail.com');>');>> >> >> wrote: >> >> >> > Umm there was a thread several months ago about how it is used on >> >> >> several >> >> >> > of the slave dbs, if I recall. >> >> >> > >> >> >> >> >> >> Indeed, you're looking for "mar
Re: [Wikitech-l] [Labs-l] Maria DB
Keeping debug symbols in binaries will result in poor performance, or it should On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman wrote: > For most projects, I recommend using the official packages available via > the MariaDB projects own apt repo. > > The official packages are based on the Debian mysql packaging where > installing the server package also installs a default database created > around generic config defaults, a debian mysql maintenance user with a > randomly generated password, and scripts (including init) that assume > privileged access via that user. That is, installing the packages provides > you with a fresh running working database with generic defaults suitable > for a small server, and certain admin tasks automated. I think that's what > the average labs and general users wants and expects. > > The packages I've built for production use at wmf strips out all of the > debianisms, the debian project script rewrites, the pre/post install > actions. They also leave debug symbols in the binaries and have compiler > flag tweaks, but do not at this stage contain any source > patches. Installing the server package doesn't create a default db, or > provide an environment where you can even start the server on a fresh sever > install without further work. Probably not a good choice for most labs > users. > > On Wednesday, February 13, 2013, Petr Bena wrote: > > > thanks for updates. > > > > Can you tell me what is a difference between maria db you are using and > > the version that is recommended for use on ubuntu? > > > > > > On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman > > 'cvml', 'afeld...@wikimedia.org');> > > > wrote: > > > >> The production migration to MariaDB was paused for a time by the EQIAD > >> datacenter migration and issues involving other projects that took up my > >> time, but the trial production roll-out will resume this month. All > signs > >> still point to our using it in production. > >> > >> I did a lot of query testing on an enwiki MariaDB 5.5 slave over the > >> course > >> of more than a month before the first production deployment. Major > >> version > >> migrations with mysql and derivatives are not to be taken lightly in > >> production environments. At a minimum, one must be concerned about > query > >> optimizer changes making one particular query type significantly slower. > >> In the case of the switch to 5.5, there are several default behavior > >> changes over 5.1 that can break applications or change results. Hence, > >> some serious work over a plodding time frame before that first > production > >> slave switch. > >> > >> Despite those efforts, a couple weeks after the switch, I saw a query > >> generated by what seems to be a very rare edge case from that AFTv4 > >> extension that violated stricter enforcement of unsigned integer types > in > >> 5.5, breaking replication and requiring one off rewriting and execution > of > >> the query locally to ensure data consistency before skipping over it. I > >> opened a bug, Mathias fixed the extension, and I haven't seen any other > >> compatibility issues from AFTv4 or anything else deployed on enwiki. > >> > >> That said, other projects utilize different extensions, so all of my > >> testing that has gone into enwiki cannot be assumed to fully cover > >> everything else. Because of that, and because I want to continue > >> proceeding with caution for all of our projects, this will continue to > be > >> a > >> slow and methodical process at this stage. Bugs in extensions that > aren't > >> used by English Wikipedia may be found and require fixing along the way. > >> > >> As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. > >> > >> Best, > >> Asher > >> > >> On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena > >> 'cvml', 'benap...@gmail.com');>> > >> wrote: > >> > >> > Okay - so what is outcome? Should we migrate beta cluster? Are we > going > >> to > >> > use it in production? > >> > > >> > > >> > On Wed, Feb 13, 2013 at 2:08 PM, Chad > >> > 'cvml', 'innocentkil...@gmail.com');>> > >> wrote: > >> > > >> >> On Wed, Feb 13, 2013 at 8:05 AM, bawolff > >> >> 'cvml', 'bawolff%2...@gmail.com');>> > >> wrote: > >> >> > Umm there was a thread several months ago about how it is used on > >> >> several > >> >> > of the slave dbs, if I recall. > >> >> > > >> >> > >> >> Indeed, you're looking for "mariadb 5.5 in production for english > >> >> wikipedia" > >> >> > >> >> http://www.gossamer-threads.com/lists/wiki/wikitech/319925 > >> >> > >> >> -Chad > >> >> > >> >> ___ > >> >> Wikitech-l mailing list > >> >> Wikitech-l@lists.wikimedia.org >> 'Wikitech-l@lists.wikimedia.org');> > >> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l > >> >> > >> > > >> > > >> > ___ > >> > Labs-l mailing list > >> > lab...@lists.wikimedia.org >> 'lab...@lists.wikimedia.org');> > >> > https://lists.wikimedia.org/mailman/
Re: [Wikitech-l] [Labs-l] Maria DB
For most projects, I recommend using the official packages available via the MariaDB projects own apt repo. The official packages are based on the Debian mysql packaging where installing the server package also installs a default database created around generic config defaults, a debian mysql maintenance user with a randomly generated password, and scripts (including init) that assume privileged access via that user. That is, installing the packages provides you with a fresh running working database with generic defaults suitable for a small server, and certain admin tasks automated. I think that's what the average labs and general users wants and expects. The packages I've built for production use at wmf strips out all of the debianisms, the debian project script rewrites, the pre/post install actions. They also leave debug symbols in the binaries and have compiler flag tweaks, but do not at this stage contain any source patches. Installing the server package doesn't create a default db, or provide an environment where you can even start the server on a fresh sever install without further work. Probably not a good choice for most labs users. On Wednesday, February 13, 2013, Petr Bena wrote: > thanks for updates. > > Can you tell me what is a difference between maria db you are using and > the version that is recommended for use on ubuntu? > > > On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman > > > wrote: > >> The production migration to MariaDB was paused for a time by the EQIAD >> datacenter migration and issues involving other projects that took up my >> time, but the trial production roll-out will resume this month. All signs >> still point to our using it in production. >> >> I did a lot of query testing on an enwiki MariaDB 5.5 slave over the >> course >> of more than a month before the first production deployment. Major >> version >> migrations with mysql and derivatives are not to be taken lightly in >> production environments. At a minimum, one must be concerned about query >> optimizer changes making one particular query type significantly slower. >> In the case of the switch to 5.5, there are several default behavior >> changes over 5.1 that can break applications or change results. Hence, >> some serious work over a plodding time frame before that first production >> slave switch. >> >> Despite those efforts, a couple weeks after the switch, I saw a query >> generated by what seems to be a very rare edge case from that AFTv4 >> extension that violated stricter enforcement of unsigned integer types in >> 5.5, breaking replication and requiring one off rewriting and execution of >> the query locally to ensure data consistency before skipping over it. I >> opened a bug, Mathias fixed the extension, and I haven't seen any other >> compatibility issues from AFTv4 or anything else deployed on enwiki. >> >> That said, other projects utilize different extensions, so all of my >> testing that has gone into enwiki cannot be assumed to fully cover >> everything else. Because of that, and because I want to continue >> proceeding with caution for all of our projects, this will continue to be >> a >> slow and methodical process at this stage. Bugs in extensions that aren't >> used by English Wikipedia may be found and require fixing along the way. >> >> As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. >> >> Best, >> Asher >> >> On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena >> > >> wrote: >> >> > Okay - so what is outcome? Should we migrate beta cluster? Are we going >> to >> > use it in production? >> > >> > >> > On Wed, Feb 13, 2013 at 2:08 PM, Chad >> > > > 'innocentkil...@gmail.com');>> >> wrote: >> > >> >> On Wed, Feb 13, 2013 at 8:05 AM, bawolff >> >> > >> 'bawolff%2...@gmail.com');>> >> wrote: >> >> > Umm there was a thread several months ago about how it is used on >> >> several >> >> > of the slave dbs, if I recall. >> >> > >> >> >> >> Indeed, you're looking for "mariadb 5.5 in production for english >> >> wikipedia" >> >> >> >> http://www.gossamer-threads.com/lists/wiki/wikitech/319925 >> >> >> >> -Chad >> >> >> >> ___ >> >> Wikitech-l mailing list >> >> Wikitech-l@lists.wikimedia.org > 'Wikitech-l@lists.wikimedia.org');> >> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> >> >> > >> > >> > ___ >> > Labs-l mailing list >> > lab...@lists.wikimedia.org > 'lab...@lists.wikimedia.org');> >> > https://lists.wikimedia.org/mailman/listinfo/labs-l >> > >> > >> ___ >> Wikitech-l mailing list >> Wikitech-l@lists.wikimedia.org > 'Wikitech-l@lists.wikimedia.org');> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> > > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
thanks for updates. Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu? On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman wrote: > The production migration to MariaDB was paused for a time by the EQIAD > datacenter migration and issues involving other projects that took up my > time, but the trial production roll-out will resume this month. All signs > still point to our using it in production. > > I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course > of more than a month before the first production deployment. Major version > migrations with mysql and derivatives are not to be taken lightly in > production environments. At a minimum, one must be concerned about query > optimizer changes making one particular query type significantly slower. > In the case of the switch to 5.5, there are several default behavior > changes over 5.1 that can break applications or change results. Hence, > some serious work over a plodding time frame before that first production > slave switch. > > Despite those efforts, a couple weeks after the switch, I saw a query > generated by what seems to be a very rare edge case from that AFTv4 > extension that violated stricter enforcement of unsigned integer types in > 5.5, breaking replication and requiring one off rewriting and execution of > the query locally to ensure data consistency before skipping over it. I > opened a bug, Mathias fixed the extension, and I haven't seen any other > compatibility issues from AFTv4 or anything else deployed on enwiki. > > That said, other projects utilize different extensions, so all of my > testing that has gone into enwiki cannot be assumed to fully cover > everything else. Because of that, and because I want to continue > proceeding with caution for all of our projects, this will continue to be a > slow and methodical process at this stage. Bugs in extensions that aren't > used by English Wikipedia may be found and require fixing along the way. > > As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. > > Best, > Asher > > On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena wrote: > > > Okay - so what is outcome? Should we migrate beta cluster? Are we going > to > > use it in production? > > > > > > On Wed, Feb 13, 2013 at 2:08 PM, Chad wrote: > > > >> On Wed, Feb 13, 2013 at 8:05 AM, bawolff wrote: > >> > Umm there was a thread several months ago about how it is used on > >> several > >> > of the slave dbs, if I recall. > >> > > >> > >> Indeed, you're looking for "mariadb 5.5 in production for english > >> wikipedia" > >> > >> http://www.gossamer-threads.com/lists/wiki/wikitech/319925 > >> > >> -Chad > >> > >> ___ > >> Wikitech-l mailing list > >> Wikitech-l@lists.wikimedia.org > >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l > >> > > > > > > ___ > > Labs-l mailing list > > lab...@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/labs-l > > > > > ___ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] [Labs-l] Maria DB
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production. I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch. Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki. That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way. As the MariaDB roll-out proceeds, I will provide updates on wikitech-l. Best, Asher On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena wrote: > Okay - so what is outcome? Should we migrate beta cluster? Are we going to > use it in production? > > > On Wed, Feb 13, 2013 at 2:08 PM, Chad wrote: > >> On Wed, Feb 13, 2013 at 8:05 AM, bawolff wrote: >> > Umm there was a thread several months ago about how it is used on >> several >> > of the slave dbs, if I recall. >> > >> >> Indeed, you're looking for "mariadb 5.5 in production for english >> wikipedia" >> >> http://www.gossamer-threads.com/lists/wiki/wikitech/319925 >> >> -Chad >> >> ___ >> Wikitech-l mailing list >> Wikitech-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> > > > ___ > Labs-l mailing list > lab...@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/labs-l > > ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
