Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Saturday, 24 August 2013 at 07:50:00 UTC, Joseph Rushton Wakeling wrote: I proposed something along these lines shortly after DConf: http://forum.dlang.org/thread/mailman.47.1369319426.13711.digitalmar...@puremagic.com I thought it could be useful both as a stability tester _and_ as a means to evaluate the prospective impact of deliberately breaking changes. It was quite well received as an idea but is probably a big job to take on ... :-( I do want to contribute one once I decide what I want to do about a more powerful server (such suite is a bit too hard for my small VPS). Pretty sure something can be done this year, but will take some time (months+).
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On 25/08/13 21:57, Dicebot wrote: I do want to contribute one once I decide what I want to do about a more powerful server (such suite is a bit too hard for my small VPS). Pretty sure something can be done this year, but will take some time (months+). Is this something where we can do some crowdfunding? Might be worth trying to invest in some server/cloud infrastructure for D projects like this that have collective benefit.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Sunday, 25 August 2013 at 20:02:49 UTC, Joseph Rushton Wakeling wrote: On 25/08/13 21:57, Dicebot wrote: I do want to contribute one once I decide what I want to do about a more powerful server (such suite is a bit too hard for my small VPS). Pretty sure something can be done this year, but will take some time (months+). Is this something where we can do some crowdfunding? Might be worth trying to invest in some server/cloud infrastructure for D projects like this that have collective benefit. Well, I wanted to do the move anyway for my own needs so it is just a planned side effect. It is not that costly, I simply don't have time to do the move properly.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On 23/08/13 20:38, H. S. Teoh wrote: One idea that occurred to me is to put large external projects under a separate tester, not bound to the core dmd/druntime/phobos autotesting, but an independent tester that regularly checks out git HEAD and compiles tests said large projects. I proposed something along these lines shortly after DConf: http://forum.dlang.org/thread/mailman.47.1369319426.13711.digitalmar...@puremagic.com I thought it could be useful both as a stability tester _and_ as a means to evaluate the prospective impact of deliberately breaking changes. It was quite well received as an idea but is probably a big job to take on ... :-(
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
13.08.2013 21:49, glycerine пишет: Grrr... Apparently nobody has been testing the D - Apache Thrift bindings since 2.061, and dmd has since accumulated multiple regressions that affect the correctness of the Thrift implementation. I emailed with David N. and he said that this was quite common for each release of dmd, and that while he used to religously evaluate each new dmd release on the Thrift bindings, he had simply not had the time for more recent recents to test each thoroughly. Serialization: this is fundamental. This really isn't the kind of thing that should ever be allowed to break. Hence it really isn't something that should be tested manually. It should be an automatic part of the automatic regression detection test suite. Where is the regression suite for D located, and how can I add to it? There used to be github issue tracking, but I don't see it any more... is it hiding under their new interface perhaps...? Thanks. - glycerine By the way, the ability to add costume projects to D autotester is already proposed without any response: http://forum.dlang.org/thread/kqm4ta$m7f$1...@digitalmars.com -- Денис В. Шеломовский Denis V. Shelomovskij
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On 8/23/2013 10:34 AM, Denis Shelomovskij wrote: By the way, the ability to add costume projects to D autotester is already proposed without any response: http://forum.dlang.org/thread/kqm4ta$m7f$1...@digitalmars.com The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread std.serialization: pre-voting review / discussion. Here's the message: - On 8/18/2013 9:33 AM, David Nadlinger wrote: Having a system that regularly, automatically runs the test suites of several larger, well-known D projects with the results being readily available to the DMD/druntime/Phobos teams would certainly help. But it's also not ideal, since if a project starts to fail, the exact nature of the issue (regression in DMD or bug in the project, and if the former, a minimal test case) can often be hard to track down for somebody not already familiar with the code base. That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Fri, Aug 23, 2013 at 11:07:35AM -0700, Walter Bright wrote: On 8/23/2013 10:34 AM, Denis Shelomovskij wrote: By the way, the ability to add costume projects to D autotester is already proposed without any response: http://forum.dlang.org/thread/kqm4ta$m7f$1...@digitalmars.com The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread std.serialization: pre-voting review / discussion. Here's the message: - On 8/18/2013 9:33 AM, David Nadlinger wrote: Having a system that regularly, automatically runs the test suites of several larger, well-known D projects with the results being readily available to the DMD/druntime/Phobos teams would certainly help. But it's also not ideal, since if a project starts to fail, the exact nature of the issue (regression in DMD or bug in the project, and if the former, a minimal test case) can often be hard to track down for somebody not already familiar with the code base. That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred. One idea that occurred to me is to put large external projects under a separate tester, not bound to the core dmd/druntime/phobos autotesting, but an independent tester that regularly checks out git HEAD and compiles tests said large projects. The devs can then monitor the status of these tests independently, and when something goes wrong, they can ping whoever is responsible for that project to investigate what might be the cause. If it's caused by the latest git commit(s), they can file regression bugs on the bugtracker, otherwise, they update their code to work with the new compiler. If the responsible person doesn't respond, or there is no contact person, we could use this forum as a kind of community notice that something needs to be fixed somewhere. If nobody responds, the project is likely not worth keeping up with. This way we don't slow down development / autotesting unnecessarily, and still let the community know when there might be potential problems with existing code. (It probably also helps for the core devs to be aware of potential regressions without being held back from code changes.) If it's important, *somebody* will step up and file bugs and/or fix the issue. If nobody cares, then probably it's not worth fretting over. T -- Do not reason with the unreasonable; you lose by definition.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On 8/23/13, H. S. Teoh hst...@quickfur.ath.cx wrote: The devs can then monitor the status of these tests independently, and when something goes wrong, they can ping whoever is responsible for that project to investigate what might be the cause. That's basically the same thing I said in that other thread. Considering that we have a very close D community, this sort of workflow could work.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
Am 23.08.2013 20:07, schrieb Walter Bright: On 8/23/2013 10:34 AM, Denis Shelomovskij wrote: By the way, the ability to add costume projects to D autotester is already proposed without any response: http://forum.dlang.org/thread/kqm4ta$m7f$1...@digitalmars.com The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread std.serialization: pre-voting review / discussion. Here's the message: - On 8/18/2013 9:33 AM, David Nadlinger wrote: Having a system that regularly, automatically runs the test suites of several larger, well-known D projects with the results being readily available to the DMD/druntime/Phobos teams would certainly help. But it's also not ideal, since if a project starts to fail, the exact nature of the issue (regression in DMD or bug in the project, and if the former, a minimal test case) can often be hard to track down for somebody not already familiar with the code base. That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred. I find it funny how hard you try to get D production ready and make (in my opinion) bad decisions affecting the future of D, but I hit every release at least 3 regressions, 1 of these is mostly a real WTF. At least run this test-suite consisting of 3rd party projects every now and then and especially before a release. I personally don't mind these regressions (well I do, but I can live with them), trying to find the root for a day or two and fix it there, but companies will. While we're at it, the D release-zip should also be checked, not beeing able to install a new release because the zip is fucked up doesn't speak for D (especially when it takes forever to download dmd for windows and you find a shitton of binaries for other OSes as well) /rant
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On 8/23/2013 3:11 PM, David wrote: I find it funny how hard you try to get D production ready and make (in my opinion) bad decisions affecting the future of D, but I hit every release at least 3 regressions, 1 of these is mostly a real WTF. Please join us in the beta test program, then. The point of it is so that users can compile their projects and find problems before we do a release.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Friday, 23 August 2013 at 20:13:21 UTC, Andrej Mitrovic wrote: On 8/23/13, H. S. Teoh hst...@quickfur.ath.cx wrote: The devs can then monitor the status of these tests independently, and when something goes wrong, they can ping whoever is responsible for that project to investigate what might be the cause. That's basically the same thing I said in that other thread. Considering that we have a very close D community, this sort of workflow could work. Yep. I don't think anybody is suggesting to add external projects to the main CI system – at least I certainly wouldn't, being part of both the compiler and library writer camps. But it could still be useful to have an automated real world health check that is available publicly, maybe even integrated with the official D package repository, if we are going to have one anytime soon (btw, did I miss any official announcement regarding code.dlang.org/dub?). This is not about shifting the responsibility for maintaining the library to the core team, it's about empowering people working on the frontend/druntime/Phobos to know when they are breaking real-world code. I think it's pretty clear at this point that the regression test suite is not anywhere near exhaustive for even the specified parts of the language, and then there are still the many holes in the spec to consider as well. Maybe we can pull this off with an improved beta process where all D users are actually persuaded to participate. But seeing that Walter seems to avoid incorporating any of the ideas in that direction that have been brought up on dmd-beta previously (more public announcements, at least a rough tentative release schedule, meaningful file names/version info, …), I'm not sure what the plan is there. David
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Sat, Aug 24, 2013 at 01:18:14AM +0200, David Nadlinger wrote: [...] Maybe we can pull this off with an improved beta process where all D users are actually persuaded to participate. But seeing that Walter seems to avoid incorporating any of the ideas in that direction that have been brought up on dmd-beta previously (more public announcements, at least a rough tentative release schedule, meaningful file names/version info, …), I'm not sure what the plan is there. [...] It could be as simple as announcing the availability of a beta release on the main D forum / newsgroup (i.e. here). I didn't even know dmd-beta existed until more than a year after I joined the D community. And now that I'm subscribed, I hardly ever hear anything from it. Posting the beta announcement here instead would, at the very least, reach a far wider audience. (Or maybe post in both places. Wouldn't hurt. The whole point of a beta *release* is for the world to know about it so that it can try things out before the actual release. No point hiding the announcement in some obscure isolated corner.) T -- Spaghetti code may be tangly, but lasagna code is just cheesy.
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Fri, Aug 23, 2013 at 11:07 AM, Walter Bright newshou...@digitalmars.comwrote: That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. AFAIR both Clang and GCC have entire third-party projects in their test suite. I know that at least SQLite is part of both, and that's a pretty big project. If I recall correctly, GCC releases are blocked on successfully compiling the Linux kernel, all of Firefox and I possibly Qt. The third-party project tests need to finish without failures as well. My recollection is a bit vague here though. Now, do they compile and run all the tests for these projects on every commit or make sure nothing has broken just before making a new release, I don't know. But I do know that it's the latter at the very least.
D-thrift package detects regressions since 2.061, where is the regression suite located?
Grrr... Apparently nobody has been testing the D - Apache Thrift bindings since 2.061, and dmd has since accumulated multiple regressions that affect the correctness of the Thrift implementation. I emailed with David N. and he said that this was quite common for each release of dmd, and that while he used to religously evaluate each new dmd release on the Thrift bindings, he had simply not had the time for more recent recents to test each thoroughly. Serialization: this is fundamental. This really isn't the kind of thing that should ever be allowed to break. Hence it really isn't something that should be tested manually. It should be an automatic part of the automatic regression detection test suite. Where is the regression suite for D located, and how can I add to it? There used to be github issue tracking, but I don't see it any more... is it hiding under their new interface perhaps...? Thanks. - glycerine
Re: D-thrift package detects regressions since 2.061, where is the regression suite located?
On Tuesday, August 13, 2013 19:49:51 glycerine wrote: Grrr... Apparently nobody has been testing the D - Apache Thrift bindings since 2.061, and dmd has since accumulated multiple regressions that affect the correctness of the Thrift implementation. I emailed with David N. and he said that this was quite common for each release of dmd, and that while he used to religously evaluate each new dmd release on the Thrift bindings, he had simply not had the time for more recent recents to test each thoroughly. Serialization: this is fundamental. This really isn't the kind of thing that should ever be allowed to break. Hence it really isn't something that should be tested manually. It should be an automatic part of the automatic regression detection test suite. Where is the regression suite for D located, and how can I add to it? We do not include 3rd party libraries or projects in any kind of regression suite, so if that's what you're looking for, you're out of luck. David or someone else working on the Thrift stuff would have had to set something up for the D Thrift stuff specifically. We do have an autotester which tests that the current compiler and standard library implementation pass their tests, as well as a tester which tests that pull requests pass the tests as well, and that can be found here: http://d.puremagic.com/test-results/ If you report bugs in bugzilla, then when they are fixed, unit tests for those bugs will be added for them so that they won't fail again. Our bugzilla can be found here: http://d.puremagic.com/issues. There used to be github issue tracking, but I don't see it any more... is it hiding under their new interface perhaps...? We've never used github issue tracking for either the compiler or D's standard libraries. Maybe they were enabled before, but if so, they were ignored. I don't know what the D Thrift stuff does though. - Jonathan M Davis
