Re: [O] Re: Test framework needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 31/03/11 05:40, Eric Schulte wrote: > Suvayu Ali writes: > >> On Wed, 30 Mar 2011 15:42:19 -0600 >> "Eric Schulte" wrote: >> This suite should actually be updated with effectively each patch which introduces new features and run after each patch. >>> >>> Agreed, in a perfect world... >>> So is it only necessary to add meat to this framework? >>> >>> Yes, I believe the best way forward would be to add tests to the >>> existing framework. >> >> I have a possibly completely useless idea regarding "automatically" >> checking regressions. As far I understand the problem now is its not >> very feasible to do automated tests with what ever test suite we have >> (or will have after the improvements) and see the exported results for >> each patch, as at some step it involves human intervention (as in, was >> the export good). Good or not good is subjective - but *consistent* is not - and consistency is important. Even if I do not like the default LaTeX output from org, I can tweak it, but there is a problem if there are unexpected changes in the export, which break my customizations or make it difficult to recreate old documents, especially if these changes are not documented. >> > > I would disagree that we need user interaction in the test suite. There > are already fully automated tests which (e.g., export to some backend > like html or tex and then programatically check for properties of the > exported results. Exactly - the tests is R work this way: you have some code which is executed, and then the resulting *output* is redirected into a file. these results are then compared to a reference output, and if they are not *identical* an error is raised. Similar could be done in org: export to LaTeX should always result in the same output, unless a change is intended (e.g. additional headers, improvements, ...). So one could compare the resulting .tex file with a reference .tex file for this test automatically, without user intervention. > > It is certainly likely that I am missing something, but I can't think of > a situation or a feature of Org-mode which could not be tested under the > current setup (mainly due to the fact that *every* user action in Emacs > reduces to a series of function calls which could be programatically > recreated). > >> >> So maybe we can have a directory on the Worg website (not part of the >> Worg git repo) where every week or so the test suites will publish with >> what ever the org-mode head is at the time for all the supported >> formats. Then us "puny lisp illiterate" users can check up on it over >> the course of the week and report back to the list if there is a >> problem. >> >> Since this way people can look at the export formats they are >> interested in, none of the formats get treated like a step child >> either. Would that be feasible? Or did I completely misunderstand the >> problem at hand? > > I'd think that a better way for contributing to the test suite in a > non-lisp manner would be to submit test cases, e.g. "this block of > Org-mode text should export to this but sometimes instead exports to > this", or "when I press this key sequence in this place in this org-mode > text I expect x to happen to the text". Correct - this is what we would, in addition to programmatic tests of individual functions, need. I would actually say that the exports / tangling / agendas / ... are the possibly the more important test cases, as they 1) only show in a later stage of ones project, and 2) errors in functions are easily detected by users and reported - and fizxed quite quickly and finally 3) I guess an export / ... includes quite a lot of functions, which are therefore tested as well (kind off...). > > We could even potentially leverage the existing Emacs macro system to > build a *record* method so that users could semi-automatically record > their actions allowing an interactive method of recording tests (or > submitting a re-creatable bug report). Or at least recording enough > information so that someone with a little bit more elisp-fu could wrap > the recorded actions into a unit test. That would be brilliant. Like the error reporting: atach the current buffer, record what was done and *the individual configuration of org / emacs* and finally email / upload it to an address, where it is automatically added to other submitted test cases, might bring us a long way closer to an very useful test base. I am actually ot aware of any other test framework, which let's "normal" users submit test cases via email / internet - I think that would be a very useful addition. > > Hope this is helpful -- Eric Most definitely, Rainer > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Aven
Re: [O] Re: Test framework needed
Suvayu Ali writes: > On Wed, 30 Mar 2011 15:42:19 -0600 > "Eric Schulte" wrote: > >> > >> > This suite should actually be updated with effectively each patch >> > which introduces new features and run after each patch. >> > >> >> Agreed, in a perfect world... >> >> > >> > So is it only necessary to add meat to this framework? >> > >> >> Yes, I believe the best way forward would be to add tests to the >> existing framework. > > I have a possibly completely useless idea regarding "automatically" > checking regressions. As far I understand the problem now is its not > very feasible to do automated tests with what ever test suite we have > (or will have after the improvements) and see the exported results for > each patch, as at some step it involves human intervention (as in, was > the export good). > I would disagree that we need user interaction in the test suite. There are already fully automated tests which (e.g., export to some backend like html or tex and then programatically check for properties of the exported results. It is certainly likely that I am missing something, but I can't think of a situation or a feature of Org-mode which could not be tested under the current setup (mainly due to the fact that *every* user action in Emacs reduces to a series of function calls which could be programatically recreated). > > So maybe we can have a directory on the Worg website (not part of the > Worg git repo) where every week or so the test suites will publish with > what ever the org-mode head is at the time for all the supported > formats. Then us "puny lisp illiterate" users can check up on it over > the course of the week and report back to the list if there is a > problem. > > Since this way people can look at the export formats they are > interested in, none of the formats get treated like a step child > either. Would that be feasible? Or did I completely misunderstand the > problem at hand? I'd think that a better way for contributing to the test suite in a non-lisp manner would be to submit test cases, e.g. "this block of Org-mode text should export to this but sometimes instead exports to this", or "when I press this key sequence in this place in this org-mode text I expect x to happen to the text". We could even potentially leverage the existing Emacs macro system to build a *record* method so that users could semi-automatically record their actions allowing an interactive method of recording tests (or submitting a re-creatable bug report). Or at least recording enough information so that someone with a little bit more elisp-fu could wrap the recorded actions into a unit test. Hope this is helpful -- Eric
Re: [O] Re: Test framework needed
On Wed, 30 Mar 2011 15:42:19 -0600 "Eric Schulte" wrote: > > > > This suite should actually be updated with effectively each patch > > which introduces new features and run after each patch. > > > > Agreed, in a perfect world... > > > > > So is it only necessary to add meat to this framework? > > > > Yes, I believe the best way forward would be to add tests to the > existing framework. I have a possibly completely useless idea regarding "automatically" checking regressions. As far I understand the problem now is its not very feasible to do automated tests with what ever test suite we have (or will have after the improvements) and see the exported results for each patch, as at some step it involves human intervention (as in, was the export good). So maybe we can have a directory on the Worg website (not part of the Worg git repo) where every week or so the test suites will publish with what ever the org-mode head is at the time for all the supported formats. Then us "puny lisp illiterate" users can check up on it over the course of the week and report back to the list if there is a problem. Since this way people can look at the export formats they are interested in, none of the formats get treated like a step child either. Would that be feasible? Or did I completely misunderstand the problem at hand? -- Suvayu Open source is the future. It sets us free.
Re: [O] Re: Test framework needed
Rainer M Krug writes: > On 30/03/11 16:18, Christian Egli wrote: >> Rainer M Krug writes: >> http://www.emacswiki.org/emacs/UnitTesting >>> >>> Am I right in assuming, that all of the possible test frameworks would >>> require org files and the expected output (tengle, export to ..., >>> agenda, ...)? In this case, would it make sense to start collecting >>> those, as they can easily be user contributed, consequently representing >>> a cross section of the use cases (even not intended use cases)? >> >> Before you go too far with this; Orgmode already contains a unit test >> suite. Look at the README in the testing directory >> (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) >> > > But it does not look as if it is used very often... There are not many > test org files, and I did not se anything which compares the resulting > exported / tangle file with an expected output? > > Please correct me if I am missing something. > You are correct that the existing test suite needs some attention and some more tests. Just a general commitment to convert problem reports from the mailing list into unit tests should be a step in the right direction. However the existing test suite (while under populated) is the result of multiple previous discussion of this nature on the mailing list, and I think that abandoning the existing infrastructure would be a step in the wrong direction. > > This suite should actually be updated with effectively each patch which > introduces new features and run after each patch. > Agreed, in a perfect world... > > So is it only necessary to add meat to this framework? > Yes, I believe the best way forward would be to add tests to the existing framework. Best -- Eric > > Rainer
Re: [O] Re: Test framework needed
Aankhen writes: > On Wed, Mar 30, 2011 at 20:43, Manuel Giraud > wrote: >> Rainer M Krug writes: >> [snip] >> >>> Please correct me if I am missing something. >>> >>> This suite should actually be updated with effectively each patch which >>> introduces new features and run after each patch. >> >> Which renders this framework far less automatic. I think that having a >> set of org files against which one could try any export and *see* that >> the results are almost correct would be enough. > I think an "automatic" solution would involve running a batch Emacs process which loads up Org-mode and then loads and runs the existing test suite. This would be both easily automatable and would allow testing of various versions of Emacs with relative ease. Best -- Eric
Re: [O] Re: Test framework needed
On Wed, Mar 30, 2011 at 20:43, Manuel Giraud wrote: > Rainer M Krug writes: > [snip] > >> Please correct me if I am missing something. >> >> This suite should actually be updated with effectively each patch which >> introduces new features and run after each patch. > > Which renders this framework far less automatic. I think that having a > set of org files against which one could try any export and *see* that > the results are almost correct would be enough. I think the “automated” part refers to running the tests. What you suggest—having a set of files that you could manually export to verify the results—wouldn’t be of much help, IMHO. First, it’d require a lot more time than executing a single command and checking the summary at the end. Second, it’d be very error-prone. A comprehensive automated test suite gives people writing patches an easier way to perform regression testing and catch any unintended consequences. On the other hand, it /does/ take a lot of effort to keep it in sync with the codebase… maybe we need a Test Fairy. ;-) Aankhen
Re: [O] Re: Test framework needed
Rainer M Krug writes: > But it does not look as if it is used very often... There are not many > test org files, and I did not se anything which compares the resulting > exported / tangle file with an expected output? It could be useful to have this kind of framework for org but I think it is difficult problem when it comes to org export. For example, is removing the "validate xhtml" button from an html export make this export invalid? I guess not. > Please correct me if I am missing something. > > This suite should actually be updated with effectively each patch which > introduces new features and run after each patch. Which renders this framework far less automatic. I think that having a set of org files against which one could try any export and *see* that the results are almost correct would be enough. -- Manuel Giraud
Re: [O] Re: Test framework needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30/03/11 16:18, Christian Egli wrote: > Rainer M Krug writes: > >>> http://www.emacswiki.org/emacs/UnitTesting >> >> Am I right in assuming, that all of the possible test frameworks would >> require org files and the expected output (tengle, export to ..., >> agenda, ...)? In this case, would it make sense to start collecting >> those, as they can easily be user contributed, consequently representing >> a cross section of the use cases (even not intended use cases)? > > Before you go too far with this; Orgmode already contains a unit test > suite. Look at the README in the testing directory > (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) > But it does not look as if it is used very often... There are not many test org files, and I did not se anything which compares the resulting exported / tangle file with an expected output? Please correct me if I am missing something. This suite should actually be updated with effectively each patch which introduces new features and run after each patch. So is it only necessary to add meat to this framework? Rainer - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel:+33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: rai...@krugs.de Skype: RMkrug -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TPp4ACgkQoYgNqgF2egpBZQCfQN+g3J2+BiyKaWzh6o847mZS PR4An0aW0Di6vfgzwsJgwfK4FajL65WP =9/CW -END PGP SIGNATURE-
Re: [O] Re: Test framework needed
As a heavy Perl user, writing /automated/ tests is a large part of my dev work. I would suggest / plea / encourage that whatever framework is used can be automated. If it cannot be run as part of an automated process it is not going to be run. Also consider a set of testing platforms (emacs version, supporting versions of other .el modules, OS version, external software). There are many dependencies that org has - being able to automate this testing is a must. Just my $0.02. Brian / MidLifeXis - Original Message From: Eric Abrahamsen To: emacs-orgmode@gnu.org Sent: Wed, March 30, 2011 9:11:23 AM Subject: [O] Re: Test framework needed On Wed, Mar 30 2011, Rainer M Krug wrote: > On 30/03/11 15:46, Eric Abrahamsen wrote: >> On Wed, Mar 30 2011, Rainer M Krug wrote: >> >>> Hi >>> >>> I was bitten again from an unintended regression in org-mode, and that >>> the second time in two weeks. >>> >>> I am probably not the right person to suggest this, but I think it is >>> time to introduce a test framework for org-mode, to ensure that the >>> (without doubt useful) approach to develop org-mode does not lead to >>> regressions. >> >> This would be the page to start with, though the most likely candidate >> (Elisp Regression Testing) is only available in Emacs trunk at the >> moment… >> >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Yup, what you would need is some org source files that exercise all of the possible export options (for testing export, for example), including weird edge cases, and then ERT (if that's what we ended up using) would provide handy functions for making sure the export output matches expectations. The excellent gentleman who created the ODT exporter, whose name currently escapes me, has already created test files for his exporter—that would be a perfect place to start. Covering all of org's various functions would end up being a bit of a PITA, though you're quite right that it's an excellent idea, and will become more and more necessary. E
Re: [O] Re: Test framework needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30/03/11 16:11, Eric Abrahamsen wrote: > On Wed, Mar 30 2011, Rainer M Krug wrote: > >> On 30/03/11 15:46, Eric Abrahamsen wrote: >>> On Wed, Mar 30 2011, Rainer M Krug wrote: >>> Hi I was bitten again from an unintended regression in org-mode, and that the second time in two weeks. I am probably not the right person to suggest this, but I think it is time to introduce a test framework for org-mode, to ensure that the (without doubt useful) approach to develop org-mode does not lead to regressions. >>> >>> This would be the page to start with, though the most likely candidate >>> (Elisp Regression Testing) is only available in Emacs trunk at the >>> moment… >>> >>> http://www.emacswiki.org/emacs/UnitTesting >> >> Am I right in assuming, that all of the possible test frameworks would >> require org files and the expected output (tengle, export to ..., >> agenda, ...)? In this case, would it make sense to start collecting >> those, as they can easily be user contributed, consequently representing >> a cross section of the use cases (even not intended use cases)? > > Yup, what you would need is some org source files that exercise all of > the possible export options (for testing export, for example), including > weird edge cases, and then ERT (if that's what we ended up using) would > provide handy functions for making sure the export output matches > expectations. The excellent gentleman who created the ODT exporter, > whose name currently escapes me, has already created test files for his > exporter—that would be a perfect place to start. > > Covering all of org's various functions would end up being a bit of a > PITA, though you're quite right that it's an excellent idea, and will > become more and more necessary. So would there be a possibility of "normal org users" (if there is such a thing ...) to contribute to this? What would be needed? Any specific structure of the org files? Rainer > > E > > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel:+33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: rai...@krugs.de Skype: RMkrug -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TPJgACgkQoYgNqgF2egoWKwCeMjWgggD7JMhVTrQTHe3f7n6s VhgAn2CG25hOa1Q4RPufarreQVYlezHm =18s2 -END PGP SIGNATURE-
[O] Re: Test framework needed
Rainer M Krug writes: >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Before you go too far with this; Orgmode already contains a unit test suite. Look at the README in the testing directory (http://orgmode.org/w/?p=org-mode.git;a=blob_plain;f=testing/README.org;hb=HEAD) -- Christian Egli Swiss Library for the Blind, Visually Impaired and Print Disabled Grubenstrasse 12, CH-8045 Zürich, Switzerland
[O] Re: Test framework needed
On Wed, Mar 30 2011, Rainer M Krug wrote: > On 30/03/11 15:46, Eric Abrahamsen wrote: >> On Wed, Mar 30 2011, Rainer M Krug wrote: >> >>> Hi >>> >>> I was bitten again from an unintended regression in org-mode, and that >>> the second time in two weeks. >>> >>> I am probably not the right person to suggest this, but I think it is >>> time to introduce a test framework for org-mode, to ensure that the >>> (without doubt useful) approach to develop org-mode does not lead to >>> regressions. >> >> This would be the page to start with, though the most likely candidate >> (Elisp Regression Testing) is only available in Emacs trunk at the >> moment… >> >> http://www.emacswiki.org/emacs/UnitTesting > > Am I right in assuming, that all of the possible test frameworks would > require org files and the expected output (tengle, export to ..., > agenda, ...)? In this case, would it make sense to start collecting > those, as they can easily be user contributed, consequently representing > a cross section of the use cases (even not intended use cases)? Yup, what you would need is some org source files that exercise all of the possible export options (for testing export, for example), including weird edge cases, and then ERT (if that's what we ended up using) would provide handy functions for making sure the export output matches expectations. The excellent gentleman who created the ODT exporter, whose name currently escapes me, has already created test files for his exporter—that would be a perfect place to start. Covering all of org's various functions would end up being a bit of a PITA, though you're quite right that it's an excellent idea, and will become more and more necessary. E
Re: [O] Re: Test framework needed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30/03/11 15:46, Eric Abrahamsen wrote: > On Wed, Mar 30 2011, Rainer M Krug wrote: > >> Hi >> >> I was bitten again from an unintended regression in org-mode, and that >> the second time in two weeks. >> >> I am probably not the right person to suggest this, but I think it is >> time to introduce a test framework for org-mode, to ensure that the >> (without doubt useful) approach to develop org-mode does not lead to >> regressions. > > This would be the page to start with, though the most likely candidate > (Elisp Regression Testing) is only available in Emacs trunk at the > moment… > > http://www.emacswiki.org/emacs/UnitTesting Am I right in assuming, that all of the possible test frameworks would require org files and the expected output (tengle, export to ..., agenda, ...)? In this case, would it make sense to start collecting those, as they can easily be user contributed, consequently representing a cross section of the use cases (even not intended use cases)? Rainer > > E > > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Tel:+33 - (0)9 53 10 27 44 Cell: +27 - (0)8 39 47 90 42 Fax (SA): +27 - (0)8 65 16 27 82 Fax (D) : +49 - (0)3 21 21 25 22 44 Fax (FR): +33 - (0)9 58 10 27 44 email: rai...@krugs.de Skype: RMkrug -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2TNpwACgkQoYgNqgF2egpnWACeMFmP1p7mKsMXsELXcFSEqW99 mMkAnivsDT7zmguzphqziXzoKdx5fqz3 =AWd5 -END PGP SIGNATURE-
[O] Re: Test framework needed
On Wed, Mar 30 2011, Rainer M Krug wrote: > Hi > > I was bitten again from an unintended regression in org-mode, and that > the second time in two weeks. > > I am probably not the right person to suggest this, but I think it is > time to introduce a test framework for org-mode, to ensure that the > (without doubt useful) approach to develop org-mode does not lead to > regressions. This would be the page to start with, though the most likely candidate (Elisp Regression Testing) is only available in Emacs trunk at the moment… http://www.emacswiki.org/emacs/UnitTesting E
