Re: ENB: Converting unit tests for @shadow

[Edward K. Ream]

On Fri, Sep 10, 2021 at 8:18 AM David Szent-Györgyi 
wrote:

For data that is generated by hand or by plugins that are not centrally
> controlled, and for tests that are difficult to write, would the
> thoroughness of property-based testing be relevant? I wrote about that
> 
> some time back.


Let me give a more direct answer to your question.

The ekr-unit-test branch does not, by itself, make Leo more or less buggy.
In future, however, the new unit testing framework will make it easier to
add new unit tests.

Imo, there is no urgent need for more tests anywhere. The tests for @shadow
seem to cover all the important cases that arise in the update algorithm.
It is, however, necessary to convince oneself that the new tests are
equivalent to the old. This I have done to my own satisfaction. I could be
wrong, but weak unit tests don't change the code being tested.

The new test frame does highlight the lack of *any* unit tests for Leo's
gui code. I am not inclined to add such tests. Using Leo on a daily basis
seems sufficient, but the ongoing Qt6 bugs are troubling.

Just a few minutes ago another long-standing bug appeared. See #2169
. I'll fix this into
devel, then merge devel into ekr-unit-tests. I'm not sure a new unit test
is warranted :-)

In short, the ekr-unit-test branch is a huge simplification of Leo's code,
but I do not plan to do much more than fix bugs as they arise. But for the
first time *other* devs (hint hint) may find Leo's unit testing framework
pleasant to use. All contributions gratefully accepted!

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS05bsuatbFxS3SU0aLrhEPCHdJNMpauib2jSjk4dNtGfg%40mail.gmail.com.

Re: ENB: Converting unit tests for @shadow

[Edward K. Ream]

On Friday, September 10, 2021 at 8:18:14 AM UTC-5 David Szent-Györgyi wrote:

> For data that is generated by hand or by plugins that are not centrally 
> controlled, and for tests that are difficult to write, would the 
> thoroughness of property-based testing be relevant? I wrote about that 
>  
> some time back. 


The short answer is "I don't know" :-)

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/a486e78e-db43-46fa-895b-03d7d8eac075n%40googlegroups.com.

Re: ENB: Converting unit tests for @shadow

[David Szent-Györgyi]

For data that is generated by hand or by plugins that are not centrally 
controlled, and for tests that are difficult to write, would the 
thoroughness of property-based testing be relevant? I wrote about that 
 some 
time back. 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/2a03ea57-a506-4c1b-a6c8-ddaabe9fb464n%40googlegroups.com.

Re: ENB: Converting unit tests for @shadow

[tbp1...@gmail.com]

If you need to evaluate the structure, maybe an xslt transform would be a 
good approach.  Or building on that, a schematron test 
(https://schematron.com/document/2760.html?publicationid=).  If you don't 
know about Schematron, it uses XPATH expressions to locate parts of the 
document under test, then tests them for correctness with rules and 
assertions.  Since Leo outlines are XML files, these may be good 
approaches.  BTW, Schematron is an international standard,   ISO/IEC 19757-3 
.

On Wednesday, September 8, 2021 at 12:15:03 PM UTC-4 Edward K. Ream wrote:

> On Wed, Sep 8, 2021 at 9:49 AM tbp1...@gmail.com  
> wrote:
>
> For converting outlines, maybe some fuzzing tests as well?  Most should 
>> fail...
>>
>
> Hehe, I had to google "python fuzzy tests".  Sounds interesting, but I'll 
> leave that for later :-)
>
> *Update re the new tests*
>
> It looks like the new unit tests *are* valid. I compared traces in the 
> setup logic in two cases:
>
> 1. Running the old unit tests (in devel, from unitTest.leo)
> 2. Running the new unit tests (in ekr-unit-test, with the new test-shadow 
> command).
>
> The output looks identical. In particular, sentinel lines are as expected. 
> I must only ensure that the following two structure are identical:
>
> 1. The structure of the children of each @test node in unitTest.leo.
> 2. The structure of *nodes created* by each new unit test.
>
> A test would likely fail if this were not true, but a by-hand check looks 
> advisable.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/cbb08b9a-e45a-4c08-b42c-09cc6f9338a6n%40googlegroups.com.

Re: ENB: Converting unit tests for @shadow

[Edward K. Ream]

On Wed, Sep 8, 2021 at 9:49 AM tbp1...@gmail.com 
wrote:

For converting outlines, maybe some fuzzing tests as well?  Most should
> fail...
>

Hehe, I had to google "python fuzzy tests".  Sounds interesting, but I'll
leave that for later :-)

*Update re the new tests*

It looks like the new unit tests *are* valid. I compared traces in the
setup logic in two cases:

1. Running the old unit tests (in devel, from unitTest.leo)
2. Running the new unit tests (in ekr-unit-test, with the new test-shadow
command).

The output looks identical. In particular, sentinel lines are as expected.
I must only ensure that the following two structure are identical:

1. The structure of the children of each @test node in unitTest.leo.
2. The structure of *nodes created* by each new unit test.

A test would likely fail if this were not true, but a by-hand check looks
advisable.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS094RusXwx-1mwKGwhidRjw-VeAMuOVqHac1o-tW2jGqA%40mail.gmail.com.

Re: ENB: Converting unit tests for @shadow

[tbp1...@gmail.com]

For converting outlines, maybe some fuzzing tests as well?  Most should 
fail...

On Wednesday, September 8, 2021 at 10:21:37 AM UTC-4 Edward K. Ream wrote:

> The ekr-unit-test branch contains the work for #1766 
> : cover Leo with 
> unit tests. Work is well along, as shown by PR #2159 
> .
>
> In this Engineering Notebook post, I'll say a few words about converting 
> the unit tests for @shadow. These unit tests are arguably the most 
> important because they test the @shadow update algorithm.
>
> The unit tests are also (by far) the most difficult to translate into 
> "flat" (traditional/new) unit tests.  Indeed, the @shadow update algorithm 
> uses an *actual* Leo outline as part of its data.  So the new unit tests 
> must do likewise.
>
> The *ConvertShadowTests.convert_node* script is in convertCommands.py. 
> convert_node is part of a larger script that converts @test nodes from 
> unitTest.leo to traditional unit tests. Doing the conversion by hand is out 
> of the question. I must be free to experiment with the contents of each 
> unit test. Altering tests by hand would be tedious and way error-prone.
>
> convert_node creates nodes that pass unit tests. That's not bad, but I 
> suspect that the tests aren't testing what I think they should. The only 
> way to know for sure is to study what the existing unit tests do. I'll 
> tweak convert_node as needed to ensure that the unit tests actually do what 
> the old tests did!
>
> *Summary*
>
> The old unit tests use actual outline data. The new unit tests will 
> recreate this data.
>
> The old tests were *functional* tests, not coverage tests. For Leo's 
> @shadow code, functional tests are much more important than coverage tests. 
> However, coverage tests will act as an additional check on the unit tests.
>
> For example, suppose I had naively assumed that all was well just because 
> all the new tests passed. Coverage testing might reveal that all the new 
> tests are actually the same test!
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/5dc16e51-9f88-4af1-ac95-602c0483c095n%40googlegroups.com.

ENB: Converting unit tests for @shadow

[Edward K. Ream]

The ekr-unit-test branch contains the work for #1766 
: cover Leo with unit 
tests. Work is well along, as shown by PR #2159 
.

In this Engineering Notebook post, I'll say a few words about converting 
the unit tests for @shadow. These unit tests are arguably the most 
important because they test the @shadow update algorithm.

The unit tests are also (by far) the most difficult to translate into 
"flat" (traditional/new) unit tests.  Indeed, the @shadow update algorithm 
uses an *actual* Leo outline as part of its data.  So the new unit tests 
must do likewise.

The *ConvertShadowTests.convert_node* script is in convertCommands.py. 
convert_node is part of a larger script that converts @test nodes from 
unitTest.leo to traditional unit tests. Doing the conversion by hand is out 
of the question. I must be free to experiment with the contents of each 
unit test. Altering tests by hand would be tedious and way error-prone.

convert_node creates nodes that pass unit tests. That's not bad, but I 
suspect that the tests aren't testing what I think they should. The only 
way to know for sure is to study what the existing unit tests do. I'll 
tweak convert_node as needed to ensure that the unit tests actually do what 
the old tests did!

*Summary*

The old unit tests use actual outline data. The new unit tests will 
recreate this data.

The old tests were *functional* tests, not coverage tests. For Leo's 
@shadow code, functional tests are much more important than coverage tests. 
However, coverage tests will act as an additional check on the unit tests.

For example, suppose I had naively assumed that all was well just because 
all the new tests passed. Coverage testing might reveal that all the new 
tests are actually the same test!

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/1d72a75c-efda-45b2-a076-26571f0f2886n%40googlegroups.com.