Re: [Bitcoin-development] Bitcoin Testing Project
on 09/26/2012 01:49 AM Wladimir said the following: > I'm willing to write this. But I know these kinds of proposals always > end in a big discussion about what should be and what should not be on > bitcoin.org, however we should be a bit pragmatic here. May I suggest a page bitcoin.org/developers, that links to a wiki page of developer resources? That way there's an easy link from the main site, but the content is readily editable and expandable. signature.asc Description: OpenPGP digital signature -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
On Wednesday, September 26, 2012 11:41:13 AM Daniel F wrote: > on 09/26/2012 01:49 AM Wladimir said the following: > > I'm willing to write this. But I know these kinds of proposals always > > end in a big discussion about what should be and what should not be on > > bitcoin.org, however we should be a bit pragmatic here. > > May I suggest a page bitcoin.org/developers, that links to a wiki page > of developer resources? > That way there's an easy link from the main site, but the content is > readily editable and expandable. The front page already has wiki links. Adding a direct link to a developer resources page there would probably make sense. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Matt, Glad to have another ninja onboard :) On 25/09/2012 21:41, Matt Corallo wrote: > Although Jenkins may not be the best system, we already have > jenkins and pull-tester (which is a dumb python script I wrote to > test all incoming pull requests from github). I have never heard of jenkins before. I need to do some more digging. is this the right thing? https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin Mantis on the other hand, I know exceptionally well. I hate duplication of work/data unless absolutely necessary. I will check jenkins out (just out of interest what is it actually meant to do? the website implies framework, but not what its for) So, currently there are 4 potential places for bugs to be reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 - forum (bitcointalk...) 5? - is there still the ability to add bugs via sourceforge? Adding to this doesnt make sense. Each one of these reporting methods is for a different thing. I am not seeking to replace these (or even unify them) I am looking for software that will take testcases and bug reports against them [and allow for test campaigns]. Mantis is so flexible and industry standard and if the jenkins plugin works... then we can keep things as they are until they fit into better places. The reason I am so behind mantis as the backbone is it works with more or less anything, and can easily modded to work with whatever people are most comfortable with - however it is exceptionally powerful and has had a constant stream of workflow improvements over the past few years. > > They both run the same set of scripts, namely those at > https://github.com/TheBlueMatt/test-scripts (its pretty basic right > now, but since it is on github, I was hoping someone would find > the inspiration to add to it). I will check it out. I wrote a very basic script that wikified the changelog, and linked to the changes and created wiki pages for the testcases. have you seen the stuff I put on bettermeans? bits keep vanishing then re appearing. This is the outline of the testing that I setup for 0.7 https://secure.bettermeans.com/projects/4256/wiki > > I dont really care if we keep using jenkins, but I figure we might > as well keep all the tests in one place? Yes, I would love to unify all build testing and testcases into one place. I am still on the fence as to including unit tests into this. However I do see 3 distinct type of testcases 1 - requirements based testcases (requirements based off the current block chain rules - these are edge cases and known interoperability issues) 2 - Acceptance based testcases - these are testcases that should be run for every build. Check out the General Acceptance Tests in the wiki link for examples and testcases 3 - Testcases for reference implementations of things (like multisig - i see these working like the /test folder when you install a new perl module) These three things alone are a massive task. and they still wont cover everything. I would like to get the workflow so that people can sponsor or donate to a specific campaign (eg a new feature is implemented, people want it tested so can donate just for that campaign [developing testcases, structure, requirements, etc]) Once this is done, I will get to do some exciting stuff (like writing fuzzers, automation, etc) unfortunately I do not know python, only perl. > > Anyway, I'm all for more testing (I'm always complaining about how > we need more tests for stuff...). Nice, I love testing. I think we will get on :) And I would rather go for interoperability between testing rather than rewriting it all. Cheers, steve > > Matt > > On Tue, 2012-09-25 at 19:32 +0100, steve wrote: >> Hi All, >> >> After the failure to get any real testing done for the 0.7 >> release (all of which is my fault) I have decided to rejig >> things. >> >> I am heavily into test driven development, and I have a strong >> background in requirements management, and automation. >> >> I want to leave bettermeans behind, maybe we might be able to >> keep the voting aspect of it, and link it into mantis. >> >> So, what I have been doing over the past few weeks is developing >> a rudimentary requirements set, basic requirement tracking, tests >> to prove/stress the requirements. >> >> The next most important thing is to get release/acceptance tests >> done - these primarily focus on new stuff doesnt break old (ie >> lose a wallet, etc) and needs no special requirements. >> >> To this end I have installed various opensource applications >> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating >> the best workflow process. >> >> This was a much longer post, but decided against it. :) >> >> So, what I want to know is who wants to be a part helping me out >> with all this? I am finalising the workflow flow diagrams and >> they should be ready for inspection soon. >> >> Anyone interested in helping
Re: [Bitcoin-development] Bitcoin Testing Project
Steve, > So, currently there are 4 potential places for bugs to be reported > 1 - jenkins (and unit tests) > 2 - git > 3 - mailing list > 4 - forum (bitcointalk...) > 5? - is there still the ability to add bugs via sourceforge? Currently github is the authoritative place to report issues. When someone reports a bug on the mailing list, IRC or forum, they are generally asked to make a github issue (or, someone else makes the issue for them). Failed tests are generally also reported on github, by the pull tester. We currently have 232 issues, mostly classified into categories such as "Bug", "Improvement", "GUI", "Wallet", and so on. Also it's easy to refer to github issues in commits with #123, with automatic linking. I'm not sure it is worth the effort to move to another system (especially if you need a another login etc...). But I'm probably misunderstanding what you're trying to do. Wladimir -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/09/2012 13:49, Wladimir wrote: > Steve, Hi Wladimir, > >> So, currently there are 4 potential places for bugs to be >> reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 >> - forum (bitcointalk...) 5? - is there still the ability to add >> bugs via sourceforge? > > Currently github is the authoritative place to report issues. When > someone reports a bug on the mailing list, IRC or forum, they are > generally asked to make a github issue (or, someone else makes the > issue for them). Failed tests are generally also reported on > github, by the pull tester. excellent, that makes things much easier. > > We currently have 232 issues, mostly classified into categories > such as "Bug", "Improvement", "GUI", "Wallet", and so on. > > Also it's easy to refer to github issues in commits with #123, with > automatic linking. > > I'm not sure it is worth the effort to move to another system > (especially if you need a another login etc...). But I'm probably > misunderstanding what you're trying to do. I think you might be misunderstanding a little. I am not trying to replace the current system, I need to make sure that what I do will be compatible with it (seamlessly so for the developer). I do not want this to generate extra work for the development team. However testing is a lot more than just bug reporting, dont get me wrong bug reports are important, but so is running a testcase and that testcase passing, especially if that testcase is linked to the proof of a requirement. I am trying to develop a qa environment that is conducive to testing and will allow the testers to shine in all their glory :) and we need different tools and methodologies. Git is too developer centric to be useful for organising testing. - however there is a large amount of software that is compatible with git, so the core development team only ever need to work with git. The linking between a bug, the requirement, the fix, the retest, and updating of regression testplan is vital. So is the ability to organise testing campaigns and assigning tests, work units and test relevant docs/scripts/ideas, etc. I hope this clears things up a bit? Cheers, steve > > Wladimir > - -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/ 2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/ ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0= =iHI1 - -END PGP SIGNATURE- -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8 oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P 9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc= =yP7N -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
Running a concurrent Mantis tracker would be confusing and fragment the development pathway. We have an issue tracker; it's on github. What's being talked about here are two separate things. Jenkins is a continuous integration system. It can be configured to run the suite of unit tests, regression tests, and any kind of automated functional tests for every commit on github and every pull request. Github is our issue tracker. Github, and only github, is where new issues should be reported (unless it's security related, in which case an email should be sent to the core devs directly). Certainly developers should be responsible for making sure that regression tests for bugs they fix make it either into the unit tests or Matt's functional test repository. QA should hold them accountable for that (re-opening tickets for bugs that have been fixed but without regression tests). The other thing we're talking about is coordinated release testing--getting release candidates in the hands of actual users and making sure that issues are reported. This is something that can't be automated as the point really is to pick up on things that the testing suite missed. You sound more qualified than me for coming up with a process, but in the end discovered issues should be reported to github, the final repository of issues that hold up Gavin from doing a release. Just my 0.002BTC Mark On Wed, Sep 26, 2012 at 6:22 AM, steve wrote: > > I think you might be misunderstanding a little. I am not trying to > replace the current system, I need to make sure that what I do will be > compatible with it (seamlessly so for the developer). I do not want this > to generate extra work for the development team. > > However testing is a lot more than just bug reporting, dont get me > wrong bug reports are important, but so is running a testcase and that > testcase passing, especially if that testcase is linked to the proof > of a requirement. I am trying to develop a qa environment that is > conducive to testing and will allow the testers to shine in all their > glory :) and we need different tools and methodologies. > > Git is too developer centric to be useful for organising testing. - > however there is a large amount of software that is compatible with > git, so the core development team only ever need to work with git. > > The linking between a bug, the requirement, the fix, the retest, and > updating of regression testplan is vital. So is the ability to > organise testing campaigns and assigning tests, work units and test > relevant docs/scripts/ideas, etc. > > I hope this clears things up a bit? > > Cheers, > > steve > -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach wrote: > Certainly developers should be responsible for making sure that regression > tests for bugs they fix make it either into the unit tests or Matt's > functional test repository. QA should hold them accountable for that > (re-opening tickets for bugs that have been fixed but without regression > tests). As a goal or general principle, this is agreeable. But slavish attention to this will only get ignored. There is finite developer resources, and regression tests for certain types of bugs, like prickly P2P network interaction bugs or RPC API bugs, could potentially involve many days or weeks of coding, to sufficiently simulate the environment. The ability to easily, automatically and programmatically reproduce certain classes of bugs is simply out of reach right now, and nobody is going to shut down development to fix that problem. We should move towards this direction, yes, but bitcoin test cases are not always going to be as easy as writing (say) a compiler testcase. We can always use the help of a few good QA coders: simulating a P2P environment and checking the RPC API are two examples of very complicated problems that -can- be automated for testing... with a lot of work. -- Jeff Garzik exMULTI, Inc. jgar...@exmulti.com -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/09/2012 17:06, Mark Friedenbach wrote: > Running a concurrent Mantis tracker would be confusing and fragment > the development pathway. We have an issue tracker; it's on github. I think you misunderstand what I am proposing. QA needs more than just an issue tracker. i have yet to find any opensource software that integrates testcases, nor any method for generating testplans, nor any method for linking testcases and plans to requirements, that work with git. We will need software for this (as well as workflow software) and it is much easier to integrate this into mantis/bugzilla. They are both so much more functional than Git. both mantis and bugzilla have full two way functionality with git. > > What's being talked about here are two separate things. Jenkins is > a continuous integration system. It can be configured to run the > suite of unit tests, regression tests, and any kind of automated > functional tests for every commit on github and every pull > request. well 3 but okay. Jenkins integrates both with mantis (and therefore a testsuite, etc) and with git. I do not see why anything should be any different. again I am not trying to change any current process, just develop some new ones. > > Github is our issue tracker. Github, and only github, is where new > issues should be reported (unless it's security related, in which > case an email should be sent to the core devs directly). You will only ever receive bug reports via git. How they are entered should not be of concern. There will be no space in mantis/zilla for bugs that are not related to testcases. > > Certainly developers should be responsible for making sure that > regression tests for bugs they fix make it either into the unit > tests or Matt's functional test repository. QA should hold them > accountable for that (re-opening tickets for bugs that have been > fixed but without regression tests). I feel very strongly that developers should not do regression testing or any signoff testing on their own code. QA should do the testing. I am 50/50 if they should write the testcases. the QA process should make things easier for the dev team, not generate more work for them. > > The other thing we're talking about is coordinated release > testing--getting release candidates in the hands of actual users > and making sure that issues are reported. This is something that > can't be automated as the point really is to pick up on things that > the testing suite missed. You sound more qualified than me for > coming up with a process, but in the end discovered issues should > be reported to github, the final repository of issues that hold up > Gavin from doing a release. All the core development team will still use git. the extra software is needed by test. (And the third point was coming up with a suite of tests for 3rd party developers to test their interoperability - this will having nothing to do with git, or mantis. But the solution should be compatible with mantis/zilla) > > Just my 0.002BTC Mark > > On Wed, Sep 26, 2012 at 6:22 AM, steve wrote: >> >> I think you might be misunderstanding a little. I am not trying >> to replace the current system, I need to make sure that what I do >> will be compatible with it (seamlessly so for the developer). I >> do not want this to generate extra work for the development >> team. >> >> However testing is a lot more than just bug reporting, dont get >> me wrong bug reports are important, but so is running a testcase >> and that testcase passing, especially if that testcase is linked >> to the proof of a requirement. I am trying to develop a qa >> environment that is conducive to testing and will allow the >> testers to shine in all their glory :) and we need different >> tools and methodologies. >> >> Git is too developer centric to be useful for organising testing. >> - however there is a large amount of software that is compatible >> with git, so the core development team only ever need to work >> with git. >> >> The linking between a bug, the requirement, the fix, the retest, >> and updating of regression testplan is vital. So is the ability >> to organise testing campaigns and assigning tests, work units and >> test relevant docs/scripts/ideas, etc. >> >> I hope this clears things up a bit? >> >> Cheers, >> >> steve >> > -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYz8CAAoJEFvEB9dQFvtQjkMH/Apa95IRh21mfNIuyK8kOSdt 55tLoT9a6DFyF1IPTgjHQlPN/A0JCPy/p2rIEEL7XzWpCMu1zU8BzBNmJsxGAZJG C0ue1eDEywKNFEMPTgQdebC2MbNSfUBA6lGJ5vijQlcXKoIuiV/LS7IMYh57T4u1 6Tc/SGypGe8kBLuFTihmIGH5uFS6arNGlcGgh+HRn+O4jKiAcw06lIoKh7S9Rj5e bmkimvOfproCIZeNQfSJH1BfYZaVVsJ1ouVI7ch6ytFpKsZ622zYF0Iq3042kEEp Fyqh9pDDNTJ/dwbyFpTx0WaxZySdZfZmQOCxFCAeLaCpop/nKeUnW5fy3i0sYno= =rfHO -END PGP SIGNATURE- -
Re: [Bitcoin-development] Bitcoin Testing Project
There are test cases that can be automated. That's Jenkins, and those will be run automagically. Then there are tests that cannot be automated; things like "Does the GUI look OK on all of the platforms that we support (Windows XP/2000/Vista/7/8, Ubuntu/Debian blah with window managers foo and bar, OSX 10.5/6/7/8)." Thanks to Matt, we're doing great with automated functional test cases (can always do better, of course). We're failing on simple, boring stuff like making sure we actually run on all of the platforms that we say we run on BEFORE final release. That is where I think a QA team can add a lot of value. Steve: I'm worried you're over-designing The Process. A release acceptance test plan could be nothing more than a step-by-step checklist on a wiki page, Google Doc, or Drobox shared folder... -- -- Gavin Andresen -- How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
Re: [Bitcoin-development] Bitcoin Testing Project
On Wed, 2012-09-26 at 13:28 +0100, steve wrote: > Hi Matt, > > Glad to have another ninja onboard :) > > On 25/09/2012 21:41, Matt Corallo wrote: > > Although Jenkins may not be the best system, we already have > > jenkins and pull-tester (which is a dumb python script I wrote to > > test all incoming pull requests from github). > > I have never heard of jenkins before. I need to do some more digging. > is this the right thing? > > https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin For a mantis plugin, sure, I guess... > > Mantis on the other hand, I know exceptionally well. I hate > duplication of work/data unless absolutely necessary. I will check > jenkins out (just out of interest what is it actually meant to do? the > website implies framework, but not what its for) Jenkins currently just runs the test script after each new commit to bitcoin (and provides binaries to anyone who wants them), so its pretty basic (though jenkins has way more features than we use). The bitcoin one lives at http://jenkins.bluematt.me/ > > > > > They both run the same set of scripts, namely those at > > https://github.com/TheBlueMatt/test-scripts (its pretty basic right > > now, but since it is on github, I was hoping someone would find > > the inspiration to add to it). > > I will check it out. I wrote a very basic script that wikified the > changelog, We currently keep a changelog at https://en.bitcoin.it/wiki/Changelog (I went back and added tons of logs a while back and it got updated, though 0.7 seems to be missing...) anyway, automating that would be nice... > and linked to the changes and created wiki pages for the > testcases. Having more info on that changelog page would be nice. > have you seen the stuff I put on bettermeans? bits keep > vanishing then re appearing. I have been meaning to catch up with the various attempts at better bitcoin testing that have started up a few times, but I keep never getting around to it... > > This is the outline of the testing that I setup for 0.7 > > https://secure.bettermeans.com/projects/4256/wiki > > > > > I dont really care if we keep using jenkins, but I figure we might > > as well keep all the tests in one place? > > Yes, I would love to unify all build testing and testcases into one > place. I am still on the fence as to including unit tests into this. > However I do see 3 distinct type of testcases Even if unit tests are considered separate, having it all run in one huge test script makes it quite easy to implement new things (like pull-tester) which test some arbitrary bitcoind commit in the same way as every other tester. > 1 - requirements based testcases (requirements based off the current > block chain rules - these are edge cases and known interoperability > issues) The BitcoinjBitcoindComparisonTool.jar file which is run as a part of the test scripts tries to hit as many block acceptance edge cases as possible (I'm sure I missed a ton, but it hits a lot too). I've also been pushing alternate implementation implementors to use it to test their own implementations. > > 2 - Acceptance based testcases - these are testcases that should be > run for every build. Check out the General Acceptance Tests in the > wiki link for examples and testcases > > 3 - Testcases for reference implementations of things (like multisig - > i see these working like the /test folder when you install a new perl > module) > > These three things alone are a massive task. and they still wont cover > everything. I would like to get the workflow so that people can > sponsor or donate to a specific campaign (eg a new feature is > implemented, people want it tested so can donate just for that > campaign [developing testcases, structure, requirements, etc]) > > Once this is done, I will get to do some exciting stuff (like writing > fuzzers, automation, etc) unfortunately I do not know python, only perl. As far as I'm concerned more test cases are more test cases, it may get unwieldy to maintain, but at least we'd have more test cases :) In terms of general testing strategies, I really prefer to script it all, jenkins is quite nice in that it can have slave workers using a different OS which run their own tests and then report back to the main jenkins instance. Getting a real Windows slave to run the installer and test that thoroughly as well as basic Mac things (I know OSX uses a very different build system...) would be nice (though I dont really have time to write all those tests...) re: GUI testing is hard: I've heard Qt's unit test framework is really powerful and can even include things like click scripting and analysis of the current views (though, I agree, its still no doubt hard). Matt -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info
Re: [Bitcoin-development] Bitcoin Testing Project
On Wed, Sep 26, 2012 at 8:53 PM, Matt Corallo wrote: > Jenkins currently just runs the test script after each new commit to > bitcoin (and provides binaries to anyone who wants them), so its pretty > basic (though jenkins has way more features than we use). The bitcoin > one lives at http://jenkins.bluematt.me/ Jenkins is excellent at cycling through tests, while additional external tools may bring some value they're not required. It's also essential to automate all tests that we really care are run— with our small active development group and volunteer contributors the only tests we can count on being run are the automated ones. Automated tests included with the software— or at least the source— are also the only way to have a good chance of catching gnarly platform interactions. I think more than talking about testing I think we need is actual testing. Code coverage from the current tests (e.g. bitcoin-test and a testnet sync) is very unimpressive, and while coverage isn't some magical silver bullet and does not, by itself, mean the tests are good flaws in uncovered code can't be detected by the tests. We also lack simple testing cycle documentation for people interested in testing manually to walk through, etc. I think all the meta discussion is not very useful until we actually have more substance to put into it. Otherwise I fear we're just building an airport by painting stripes and waiting for the planes to land... If someone wants to help and would like a list of some of the testing I think would be useful, ping me off-list and I can blast some suggestions. But I assume that anyone who actually wants to work on this isn't short of ideas, and at this point "work on what interests you, report what interesting thing you accomplish or discover" is probably a perfectly fine level of coordination. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
