Re: [SMW-devel] Testing SMW

[James HK]

Hi,

> for direction on how to test SMW. I don't know a thing about the automated
> testing, so I'd appreciate guidance on how to get started.

Let me try to elaborate a bit on this topic. Our automated testing
approach (as explained in the cited document you linked) contains two
varying approaches.

The first one is to use PHPUnit directly, which is to test an
individual class for its consistency with the interface or the
contract it is expected to fulfill. While this approach is the
preferred method of adding automated scripts it can be difficult and
sometimes impossible (in terms of time/effort vs. result) to reach a
conclusive implementation that would test the boundaries of a class
(this is especially the case for most legacy code we have in SMW).

> That said, there's not much said about how to manually test SMW. When you
> want testing on a new branch of SMW, do you have a checklist of basic
> functions that should be tested?

The second approach is a bit more user friendly and is based on a JSON
scripted test that generates assertions based on the content described
by a test file [0]. The advantage of this method is that those tests
are to a certain degree closer to an input a user would made by
comparing the output that is expected from either a parser or a result
printer.

For example, if you pick [1] it should be fairly obvious what is
tested and further allowing a "normal" user to copy the test case (the
data not the file) into a wiki with the expectation to replicate the
same results as specified by the test.

It is not exactly a checklist but those test follow a certain
structure and were created because of an issue a user had or they
seemed to cover a general use case.

Those tests in [0] do use a real DB connection (are not mocked and
more resources intensive as the whole MW setup is used) and are
executed on each PR against different DB settings (mysql, sqlite,
postgres, SPARQLStore etc.) which gives a good indication about the
supported platforms.

The more we have of such scripted integration tests the closer is our
predication that the system behaves as expected.

Of course, some things can't be tested which has been proven in the
past by issues that appear in tandem with MediaWiki. [2] is a typical
case that no matter how much you test it is unlikely to be caught by
any test (due to the constraints the environmental setting dictates).

PS : We don't do browser tests as that is quite tedious to setup and
run without employing to much infrastructure. Maybe running against a
webdriver [3] could be an alternative for testing browser output.

[0] 
https://github.com/SemanticMediaWiki/SemanticMediaWiki/tree/master/tests/phpunit/Integration/ByJsonScript/Fixtures

[1] 
https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/phpunit/Integration/ByJsonScript/Fixtures/p-0204%20%5B%23870%2Cen%5D%20set%20parser%20with%20error%20output.json

[2] 
https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1201#issuecomment-147389206

[3] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/201

Cheers

On 10/28/15, Jeroen De Dauw  wrote:
> Hey Daren,
>
> What exactly are you trying to achieve? Do you want to verify SMW runs
> correctly on your system? Do you want to contribute to the project by
> testing? Something else?
>
> We do not have a list of scenarios to manually test, as this is not a
> maintainable approach. Instead we have several thousand scripted tests
> which we can run automatically. If you want to contribute to the project,
> manually testing is not the way to go. With some exceptions, for which
> which we have a testers group. You can join this group and be notified when
> one of the developers wants some manual testing to be done, which will
> include a description of what to do. If you want to join this group, please
> send me your GitHub username.
>
> Cheers
>
> --
> Jeroen De Dauw - http://www.bn2vs.com
> Software craftsmanship advocate
> Developer at Wikimedia Germany
> ~=[,,_,,]:3
>

--
___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

Re: [SMW-devel] Testing SMW

[Jeroen De Dauw]

Hey Daren,

What exactly are you trying to achieve? Do you want to verify SMW runs
correctly on your system? Do you want to contribute to the project by
testing? Something else?

We do not have a list of scenarios to manually test, as this is not a
maintainable approach. Instead we have several thousand scripted tests
which we can run automatically. If you want to contribute to the project,
manually testing is not the way to go. With some exceptions, for which
which we have a testers group. You can join this group and be notified when
one of the developers wants some manual testing to be done, which will
include a description of what to do. If you want to join this group, please
send me your GitHub username.

Cheers

--
Jeroen De Dauw - http://www.bn2vs.com
Software craftsmanship advocate
Developer at Wikimedia Germany
~=[,,_,,]:3
--
___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel

Re: [SMW-devel] Testing SMW

[Daren Welsh]

Thanks Jeroen and James. I'll read through this several times and see if I
can figure out how to do some PHPUnit tests. I'm actually already added to
the SMW Testers team, but I have to admit I haven't really helped since I
haven't (yet) learned how to use PHPUnit

While I do want to contribute back to SMW directly, I'm currently focused
on a project with James Montalvo where we can install a SMW wiki farm with
VE, Elasticsearch, and all the other bells and whistles with a few simple
commands on a barebones CentOS install. Each time we add to or modify these
scripts, we have been running manual tests. We will definitely learn how to
use PHPUnit to add automated testing to our project.

Wish we could join you in Barcelona!

Daren



On Wed, Oct 28, 2015 at 8:00 AM, James HK 
wrote:

> Hi,
>
> > for direction on how to test SMW. I don't know a thing about the
> automated
> > testing, so I'd appreciate guidance on how to get started.
>
> Let me try to elaborate a bit on this topic. Our automated testing
> approach (as explained in the cited document you linked) contains two
> varying approaches.
>
> The first one is to use PHPUnit directly, which is to test an
> individual class for its consistency with the interface or the
> contract it is expected to fulfill. While this approach is the
> preferred method of adding automated scripts it can be difficult and
> sometimes impossible (in terms of time/effort vs. result) to reach a
> conclusive implementation that would test the boundaries of a class
> (this is especially the case for most legacy code we have in SMW).
>
> > That said, there's not much said about how to manually test SMW. When you
> > want testing on a new branch of SMW, do you have a checklist of basic
> > functions that should be tested?
>
> The second approach is a bit more user friendly and is based on a JSON
> scripted test that generates assertions based on the content described
> by a test file [0]. The advantage of this method is that those tests
> are to a certain degree closer to an input a user would made by
> comparing the output that is expected from either a parser or a result
> printer.
>
> For example, if you pick [1] it should be fairly obvious what is
> tested and further allowing a "normal" user to copy the test case (the
> data not the file) into a wiki with the expectation to replicate the
> same results as specified by the test.
>
> It is not exactly a checklist but those test follow a certain
> structure and were created because of an issue a user had or they
> seemed to cover a general use case.
>
> Those tests in [0] do use a real DB connection (are not mocked and
> more resources intensive as the whole MW setup is used) and are
> executed on each PR against different DB settings (mysql, sqlite,
> postgres, SPARQLStore etc.) which gives a good indication about the
> supported platforms.
>
> The more we have of such scripted integration tests the closer is our
> predication that the system behaves as expected.
>
> Of course, some things can't be tested which has been proven in the
> past by issues that appear in tandem with MediaWiki. [2] is a typical
> case that no matter how much you test it is unlikely to be caught by
> any test (due to the constraints the environmental setting dictates).
>
> PS : We don't do browser tests as that is quite tedious to setup and
> run without employing to much infrastructure. Maybe running against a
> webdriver [3] could be an alternative for testing browser output.
>
> [0]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/tree/master/tests/phpunit/Integration/ByJsonScript/Fixtures
>
> [1]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/phpunit/Integration/ByJsonScript/Fixtures/p-0204%20%5B%23870%2Cen%5D%20set%20parser%20with%20error%20output.json
>
> [2]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1201#issuecomment-147389206
>
> [3] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/201
>
> Cheers
>
> On 10/28/15, Jeroen De Dauw  wrote:
> > Hey Daren,
> >
> > What exactly are you trying to achieve? Do you want to verify SMW runs
> > correctly on your system? Do you want to contribute to the project by
> > testing? Something else?
> >
> > We do not have a list of scenarios to manually test, as this is not a
> > maintainable approach. Instead we have several thousand scripted tests
> > which we can run automatically. If you want to contribute to the project,
> > manually testing is not the way to go. With some exceptions, for which
> > which we have a testers group. You can join this group and be notified
> when
> > one of the developers wants some manual testing to be done, which will
> > include a description of what to do. If you want to join this group,
> please
> > send me your GitHub username.
> >
> > Cheers
> >
> > --
> > Jeroen De Dauw - http://www.bn2vs.com
> > Software craftsmanship advocate
> > Developer at 

Re: [SMW-devel] Testing SMW

[James HK]

Hi,

> the SMW Testers team, but I have to admit I haven't really helped since I
> haven't (yet) learned how to use PHPUnit

If you have a specific question (or problem) with PHPUnit, just drop
us a line via [0] as I'm reluctant to give (or review) code examples
via the mailing-list.

> VE, Elasticsearch, and all the other bells and whistles with a few simple
> commands on a barebones CentOS install. Each time we add to or modify these

I'm not sure how relevant [1] would be to your efforts.

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1218

Cheers

On 10/28/15, Daren Welsh  wrote:
> Thanks Jeroen and James. I'll read through this several times and see if I
> can figure out how to do some PHPUnit tests. I'm actually already added to
> the SMW Testers team, but I have to admit I haven't really helped since I
> haven't (yet) learned how to use PHPUnit
>
> While I do want to contribute back to SMW directly, I'm currently focused
> on a project with James Montalvo where we can install a SMW wiki farm with
> VE, Elasticsearch, and all the other bells and whistles with a few simple
> commands on a barebones CentOS install. Each time we add to or modify these
> scripts, we have been running manual tests. We will definitely learn how to
> use PHPUnit to add automated testing to our project.
>
> Wish we could join you in Barcelona!
>
> Daren
>
>
>
> On Wed, Oct 28, 2015 at 8:00 AM, James HK 
> wrote:
>
>> Hi,
>>
>> > for direction on how to test SMW. I don't know a thing about the
>> automated
>> > testing, so I'd appreciate guidance on how to get started.
>>
>> Let me try to elaborate a bit on this topic. Our automated testing
>> approach (as explained in the cited document you linked) contains two
>> varying approaches.
>>
>> The first one is to use PHPUnit directly, which is to test an
>> individual class for its consistency with the interface or the
>> contract it is expected to fulfill. While this approach is the
>> preferred method of adding automated scripts it can be difficult and
>> sometimes impossible (in terms of time/effort vs. result) to reach a
>> conclusive implementation that would test the boundaries of a class
>> (this is especially the case for most legacy code we have in SMW).
>>
>> > That said, there's not much said about how to manually test SMW. When
>> > you
>> > want testing on a new branch of SMW, do you have a checklist of basic
>> > functions that should be tested?
>>
>> The second approach is a bit more user friendly and is based on a JSON
>> scripted test that generates assertions based on the content described
>> by a test file [0]. The advantage of this method is that those tests
>> are to a certain degree closer to an input a user would made by
>> comparing the output that is expected from either a parser or a result
>> printer.
>>
>> For example, if you pick [1] it should be fairly obvious what is
>> tested and further allowing a "normal" user to copy the test case (the
>> data not the file) into a wiki with the expectation to replicate the
>> same results as specified by the test.
>>
>> It is not exactly a checklist but those test follow a certain
>> structure and were created because of an issue a user had or they
>> seemed to cover a general use case.
>>
>> Those tests in [0] do use a real DB connection (are not mocked and
>> more resources intensive as the whole MW setup is used) and are
>> executed on each PR against different DB settings (mysql, sqlite,
>> postgres, SPARQLStore etc.) which gives a good indication about the
>> supported platforms.
>>
>> The more we have of such scripted integration tests the closer is our
>> predication that the system behaves as expected.
>>
>> Of course, some things can't be tested which has been proven in the
>> past by issues that appear in tandem with MediaWiki. [2] is a typical
>> case that no matter how much you test it is unlikely to be caught by
>> any test (due to the constraints the environmental setting dictates).
>>
>> PS : We don't do browser tests as that is quite tedious to setup and
>> run without employing to much infrastructure. Maybe running against a
>> webdriver [3] could be an alternative for testing browser output.
>>
>> [0]
>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/tree/master/tests/phpunit/Integration/ByJsonScript/Fixtures
>>
>> [1]
>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/phpunit/Integration/ByJsonScript/Fixtures/p-0204%20%5B%23870%2Cen%5D%20set%20parser%20with%20error%20output.json
>>
>> [2]
>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/1201#issuecomment-147389206
>>
>> [3] https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/201
>>
>> Cheers
>>
>> On 10/28/15, Jeroen De Dauw  wrote:
>> > Hey Daren,
>> >
>> > What exactly are you trying to achieve? Do you want to verify SMW runs

[SMW-devel] Testing SMW

[Daren Welsh]

I found my way to
https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/README.md
for direction on how to test SMW. I don't know a thing about the automated
testing, so I'd appreciate guidance on how to get started.

That said, there's not much said about how to manually test SMW. When you
want testing on a new branch of SMW, do you have a checklist of basic
functions that should be tested?

For example:
* Set [[Property::Value]] on Page 1
* Create Page 2 with {{#ask: [[Property::+]] |?Property }} and verify "Page
1" and "Value" show up in a table
* etc

Maybe this is too difficult to conclusively answer since there are many
ways SMW can be used (and abused). I was just hoping someone had a running
checklist for when they do testing.

Daren

-- 
__
http://enterprisemediawiki.org
http://mixcloud.com/darenwelsh
http://www.beatportfolio.com
--
___
Semediawiki-devel mailing list
Semediawiki-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-devel