Proposal: moving tests to GitHub

2013-01-22 Thread Odin Hørthe Omdal 

Hi!

  We just had a small discussion on webapps-testsuite [1] about the  
possibility of moving the webapps tests.  I was wrongly under the  
impression that we had discussed this before (hey, confusion is not a  
crime ;) ).  Now that HTML has done the move, I think it is time for us to  
look at it too.  Ms2ger and Robin, which did most of the HTML testsuite  
[2] move were happy to help [3].


Ms2ger proposed merging our repository with HTML at the same time and not  
necessarily having one repository for each group.  I was already thinking  
such a move might be beneficial to do for webapps and webappsec, but it  
might be even more simple to also have html testsuite in that merge.


Robin wrote an email describing what they did for the HTML move, some of  
which should be relevant for our potential move too [4].  Of particular  
interest the possibility for having tests in folders by section of the  
spec, and making submissions pull requests.



I'm interested in such a move because it'll make our tests more visible,  
and easier to contribute to.  Just this weekend the Server Sent Events  
spec got 6 pull requests from a GitHub user "Yaffle" [5].  Note that that  
particular repository will be moved into however our new test repository  
structure will be, so it probably won't remain as a separate repository.



[1]  
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0019.html

[2] https://github.com/w3c/html-testsuite
[3]  
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0022.html
[4]  
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0024.html

[5] https://github.com/w3c/event-source/pulls

--
Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Florian Bösch 
I think it's a good idea. The WebGL specification/tests moved to github
which made contributing patches (as pull requests) a lot easier.


On Tue, Jan 22, 2013 at 11:53 AM, Odin Hørthe Omdal wrote:

> Hi!
>
>   We just had a small discussion on webapps-testsuite [1] about the
> possibility of moving the webapps tests.  I was wrongly under the
> impression that we had discussed this before (hey, confusion is not a crime
> ;) ).  Now that HTML has done the move, I think it is time for us to look
> at it too.  Ms2ger and Robin, which did most of the HTML testsuite [2] move
> were happy to help [3].
>
> Ms2ger proposed merging our repository with HTML at the same time and not
> necessarily having one repository for each group.  I was already thinking
> such a move might be beneficial to do for webapps and webappsec, but it
> might be even more simple to also have html testsuite in that merge.
>
> Robin wrote an email describing what they did for the HTML move, some of
> which should be relevant for our potential move too [4].  Of particular
> interest the possibility for having tests in folders by section of the
> spec, and making submissions pull requests.
>
>
> I'm interested in such a move because it'll make our tests more visible,
> and easier to contribute to.  Just this weekend the Server Sent Events spec
> got 6 pull requests from a GitHub user "Yaffle" [5].  Note that that
> particular repository will be moved into however our new test repository
> structure will be, so it probably won't remain as a separate repository.
>
>
> [1] http://lists.w3.org/Archives/**Public/public-webapps-**
> testsuite/2013Jan/0019.html
> [2] 
> https://github.com/w3c/html-**testsuite
> [3] http://lists.w3.org/Archives/**Public/public-webapps-**
> testsuite/2013Jan/0022.html
> [4] http://lists.w3.org/Archives/**Public/public-webapps-**
> testsuite/2013Jan/0024.html
> [5] 
> https://github.com/w3c/event-**source/pulls
>
> --
> Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software,
> http://opera.com
>
>

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 
On 1/22/13 11:53 AM, "Odin Hørthe Omdal"  wrote:

>Hi!
>
>   We just had a small discussion on webapps-testsuite [1] about the
>possibility of moving the webapps tests.  I was wrongly under the
>impression that we had discussed this before (hey, confusion is not a
>crime ;) ).

We had such a discussion in testing a while back.[a]

>Now that HTML has done the move, I think it is time for us to
>look at it too.  Ms2ger and Robin, which did most of the HTML testsuite
>[2] move were happy to help [3].
>
>Ms2ger proposed merging our repository with HTML at the same time and not
> 
>necessarily having one repository for each group.  I was already thinking
> 
>such a move might be beneficial to do for webapps and webappsec, but it
>might be even more simple to also have html testsuite in that merge.

There are benefits to both approaches. I would be in favor of having a
repository per spec (named tr_shortname-testsuite). This will make it a
lot easier to securely give scoped commit rights to external contributors
when the need arises.

>Robin wrote an email describing what they did for the HTML move, some of
>which should be relevant for our potential move too [4].  Of particular
>interest the possibility for having tests in folders by section of the
>spec, and making submissions pull requests.
>
>
>I'm interested in such a move because it'll make our tests more visible,
>and easier to contribute to.  Just this weekend the Server Sent Events
>spec got 6 pull requests from a GitHub user "Yaffle" [5].  Note that that
> 
>particular repository will be moved into however our new test repository
>structure will be, so it probably won't remain as a separate repository.

More thoughts here:
http://tobie.github.com/w3c-testing-plan/unofficial-w3c-testing-plan-201201
16.html#transition-test-repositories-to-github

--tobie

---
[a]: 
http://lists.w3.org/Archives/Public/public-test-infra/2012JulSep/0027.html

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Anne van Kesteren 
On Tue, Jan 22, 2013 at 12:11 PM, Tobie Langel  wrote:
> There are benefits to both approaches. I would be in favor of having a
> repository per spec (named tr_shortname-testsuite). This will make it a
> lot easier to securely give scoped commit rights to external contributors
> when the need arises.

Given how we relatively frequently move features around and have not
exactly figured out the overall architecture I don't think that's an
approach that scales.


-- 
http://annevankesteren.nl/

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 
On 1/22/13 12:20 PM, "Anne van Kesteren"  wrote:

>On Tue, Jan 22, 2013 at 12:11 PM, Tobie Langel  wrote:
>> There are benefits to both approaches. I would be in favor of having a
>> repository per spec (named tr_shortname-testsuite). This will make it a
>> lot easier to securely give scoped commit rights to external
>>contributors
>> when the need arises.
>
>Given how we relatively frequently move features around and have not
>exactly figured out the overall architecture I don't think that's an
>approach that scales.

That's definitely something to keep in mind. How frequent is it that a
feature moves from one spec to another (that, is outside of the continuous
flow of features that migrate from HTML5 to WebApps)?

Is your concern about history loss?

--tobie

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Anne van Kesteren 
On Tue, Jan 22, 2013 at 12:30 PM, Tobie Langel  wrote:
> That's definitely something to keep in mind. How frequent is it that a
> feature moves from one spec to another (that, is outside of the continuous
> flow of features that migrate from HTML5 to WebApps)?
>
> Is your concern about history loss?

My concern is 1) make work and 2) overhead of deciding what has to be
tested where. E.g. tests for the ProgressEvent interface test quite a
bit of IDL related requirements, where do they go? From a distance it
might all appear modular, but it really is all quite interconnected
and by creating artificial boundaries that interconnectedness might
not get testing.


-- 
http://annevankesteren.nl/

Re: Proposal: moving tests to GitHub

2013-01-22 Thread James Graham 

On 01/22/2013 12:37 PM, Anne van Kesteren wrote:

On Tue, Jan 22, 2013 at 12:30 PM, Tobie Langel  wrote:

That's definitely something to keep in mind. How frequent is it that a
feature moves from one spec to another (that, is outside of the continuous
flow of features that migrate from HTML5 to WebApps)?

Is your concern about history loss?


My concern is 1) make work and 2) overhead of deciding what has to be
tested where. E.g. tests for the ProgressEvent interface test quite a
bit of IDL related requirements, where do they go? From a distance it
might all appear modular, but it really is all quite interconnected
and by creating artificial boundaries that interconnectedness might
not get testing.


I don't have a strong opinion either way here, but I note that it is 
generally possible to move things between repos relatively easily 
without losing history e.g. using git subtree[1], so I don't think that 
it is necessary to make a decision on this before moving to git.


With that in mind, I suggest doing the move with the repository 
structure more or less as it is today and then later merging the repos 
if we decide that is desirable.


[1] https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 
On 1/22/13 12:37 PM, "Anne van Kesteren"  wrote:

>On Tue, Jan 22, 2013 at 12:30 PM, Tobie Langel  wrote:
>> That's definitely something to keep in mind. How frequent is it that a
>> feature moves from one spec to another (that, is outside of the
>>continuous
>> flow of features that migrate from HTML5 to WebApps)?
>>
>> Is your concern about history loss?
>
>My concern is 1) make work and 2) overhead of deciding what has to be
>tested where.

Figuring out precisely what it is you are trying to test is the crux of
the work. Having a repository per specification lightens the cognitive
load, not burdens it more. Especially for external contributors which
might not be aware of which WG handles what specs.

>E.g. tests for the ProgressEvent interface test quite a
>bit of IDL related requirements, where do they go?

Well, in that case[1], it seems the decision to put it under the
ProgressEvent directory has already been made. I don't see how this folder
being the root of a git repository or a subfolder is relevant.

With regards to the specifics of WebIDL requirements testing, there's work
going on to parse WebIDL and generate assertions out of it. Where that
isn't sufficient it seems a set of dedicated assertions (added to
testharness.js) would greatly simplify testing and answering the question
as to where these should go.

>From a distance it
>might all appear modular, but it really is all quite interconnected
>and by creating artificial boundaries that interconnectedness might
>not get testing.

These boundaries are created at the spec level. I see no harm in
replicating them at the test level. And there's certainly nothing
artificial in doing so.

That said, I agree integration testing is key, but it should be done
explicitly and methodically, not by relying on accidental coverage that's
hard to measure. The fun part about integration testing is it actually
tests the specs more than it tests the implementations themselves.

--tobie
 
---
[1]: http://w3c-test.org/webapps/ProgressEvents/tests/submissions/

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Odin Hørthe Omdal 

Tobie Langel wrote:

Odin wrote:
Ms2ger proposed merging our repository with HTML at the same time and  
not

necessarily having one repository for each group. I was already thinking
such a move might be beneficial to do for webapps and webappsec, but it
might be even more simple to also have html testsuite in that merge.


There are benefits to both approaches. I would be in favor of having a
repository per spec (named tr_shortname-testsuite). This will make it a
lot easier to securely give scoped commit rights to external contributors
when the need arises.


I'm not really sure if that is needed. If we can trust someone in one  
repository, why not in all?


Having one repository also makes that one more visible, and if you already  
have a checkout for fixing CORS tests, and then suddenly encounter an SSE  
error, you can easily write that test in the checkout you already have.  I  
think the best people we can optimize for is the people who have already  
written some tests.  If it's easy for them to spread out, then we'll get  
more tests.


The visibility might also help, person X forks the test repo in order to  
fix a test in Microdata.  Any linking or talking she'll do about that fix  
will point people to that repo.  If people are like me, they might  
normally poke around a bit in the tree and suddenly they'll also know  
where the tests for e.g. XMLHttpRequest is at.



But what wins me over, is really the overhead question. Do anyone really  
want to manage lots of repositories?  And for what reason?  Also, we want  
more reviewers.  If I'm already added for CORS, I could help out for say  
XMLHttpRequest if there's a submission/pull request languishing there.


Anyway, for my part, the how-to-split repository issue is not that  
important compared to having the tests at GitHub in the first place :-)


--
Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Robin Berjon 

On 22/01/2013 13:27 , Odin Hørthe Omdal wrote:

I'm not really sure if that is needed. If we can trust someone in one
repository, why not in all?


I'd add to that: the odds are that if someone is screwing things up, 
it's better to have more eyes on what they're doing.



But what wins me over, is really the overhead question. Do anyone really
want to manage lots of repositories?  And for what reason?  Also, we
want more reviewers.  If I'm already added for CORS, I could help out
for say XMLHttpRequest if there's a submission/pull request languishing
there.


I think Odin makes convincing arguments. For me it's really the outreach 
argument. Just one repo, carrying its one setup and one set of docs, can 
easily be pitched as the One True Place to Save The Web. It's a lot 
easier to explain at a conference or such: just go there, and patch stuff.



Anyway, for my part, the how-to-split repository issue is not that
important compared to having the tests at GitHub in the first place :-)


Agreed. But how about we start with just one repo and then split them 
into several if it's a problem?


--
Robin Berjon - http://berjon.com/ - @robinberjon

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 


On 1/22/13 2:23 PM, "Robin Berjon"  wrote:

>On 22/01/2013 13:27 , Odin Hørthe Omdal wrote:
>> I'm not really sure if that is needed. If we can trust someone in one
>> repository, why not in all?
>
>I'd add to that: the odds are that if someone is screwing things up,
>it's better to have more eyes on what they're doing.
>
>> But what wins me over, is really the overhead question. Do anyone really
>> want to manage lots of repositories?  And for what reason?  Also, we
>> want more reviewers.  If I'm already added for CORS, I could help out
>> for say XMLHttpRequest if there's a submission/pull request languishing
>> there.
>
>I think Odin makes convincing arguments. For me it's really the outreach
>argument. Just one repo, carrying its one setup and one set of docs, can
>easily be pitched as the One True Place to Save The Web. It's a lot
>easier to explain at a conference or such: just go there, and patch stuff.

Yes, I guess what I want to avoid at all costs is the split per WG which
has boundaries that partially happen at IP level, rather than strictly at
the technology level.

Whether we end up as:

w3c-tests/
deviceorienation/
html5/
pointerevents/
progressevent/
xmlhttprequest/

or: 

deviceorienation-tests/
html5-tests/
pointerevents-tests/
progressevent-tests/
xmlhttprequest-tests/

Doesn't really matter (though I do find the former more readable). What
bothers me however is how had to parse per-WG-organization is for
newcomers.

--tobie

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Robin Berjon 

On 22/01/2013 14:48 , Tobie Langel wrote:

Yes, I guess what I want to avoid at all costs is the split per WG which
has boundaries that partially happen at IP level, rather than strictly at
the technology level.


My understanding is that we don't have to care about spec-IP issues in 
test suites because when you contribute to a test suite you're not 
contributing to the spec's essential claims.


You *do* need to make the proper commitments for the test suite, but 
those are much lighter and can easily be extended to all.



Whether we end up as:

 w3c-tests/
 deviceorienation/
 html5/
 pointerevents/
 progressevent/
 xmlhttprequest/

or:

 deviceorienation-tests/
 html5-tests/
 pointerevents-tests/
 progressevent-tests/
 xmlhttprequest-tests/

Doesn't really matter (though I do find the former more readable). What
bothers me however is how had to parse per-WG-organization is for
newcomers.


That's why we're proposing to ditch per-WG anything here. The way 
html-testsuite is set up, we already have subdirectories for html, 
canvas2d, and microdata. Those are all from the HTML WG, but they're 
just listed as the individual specs. We can keep on adding more specs in 
there (the Web Crypto people are looking to do that).


--
Robin Berjon - http://berjon.com/ - @robinberjon

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 
On 1/22/13 4:45 PM, "Robin Berjon"  wrote:

>On 22/01/2013 14:48 , Tobie Langel wrote:
>> Yes, I guess what I want to avoid at all costs is the split per WG which
>> has boundaries that partially happen at IP level, rather than strictly
>>at
>> the technology level.
>
>My understanding is that we don't have to care about spec-IP issues in
>test suites because when you contribute to a test suite you're not
>contributing to the spec's essential claims.

That's correct.

>You *do* need to make the proper commitments for the test suite, but
>those are much lighter and can easily be extended to all.

Moving to GitHub should be an excellent occasion to revisit how the CLA
works and provide better integration, e.g.: by using something like
CLAHub[1].

>That's why we're proposing to ditch per-WG anything here. The way
>html-testsuite is set up, we already have subdirectories for html,
>canvas2d, and microdata. Those are all from the HTML WG, but they're
>just listed as the individual specs. We can keep on adding more specs in
>there (the Web Crypto people are looking to do that).

That sounds good to me. It's the per WG siloing I'm opposed to, not the
one repository to rule them all idea.

--tobie

---
[1]: https://github.com/jasonm/clahub

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Robin Berjon 

On 22/01/2013 17:14 , Tobie Langel wrote:

On 1/22/13 4:45 PM, "Robin Berjon"  wrote:

You *do* need to make the proper commitments for the test suite, but
those are much lighter and can easily be extended to all.


Moving to GitHub should be an excellent occasion to revisit how the CLA
works and provide better integration, e.g.: by using something like
CLAHub[1].


FYI we're looking at CLAHub as a possible solution for this (either 
directly or with a few modifications to tie it into our systems). No 
promises but it's on the table.



That's why we're proposing to ditch per-WG anything here. The way
html-testsuite is set up, we already have subdirectories for html,
canvas2d, and microdata. Those are all from the HTML WG, but they're
just listed as the individual specs. We can keep on adding more specs in
there (the Web Crypto people are looking to do that).


That sounds good to me. It's the per WG siloing I'm opposed to, not the
one repository to rule them all idea.


Good! Well, it looks like everyone agrees... If we're forging ahead, I 
have admin rights to the repo so you know who to prod.


--
Robin Berjon - http://berjon.com/ - @robinberjon

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Julian Aubourg 
I love the idea of moving to github.

The one-repo idea, while much simpler from a maintenance point of view,
could easily be a burden on users that subscribe to it. Even more so for
people who can merge PRs (and thus will receive an email for a PR
initiatedfor any spec).

Not saying it is blocking but it's something to keep in mind. Mail filters
can go a long way here but filtering out specific specs kinda defeats the
purpose of having so many eyes looking at everything.

Le mardi 22 janvier 2013, Robin Berjon a écrit :

> On 22/01/2013 17:14 , Tobie Langel wrote:
>
>> On 1/22/13 4:45 PM, "Robin Berjon"  wrote:
>>
>>> You *do* need to make the proper commitments for the test suite, but
>>> those are much lighter and can easily be extended to all.
>>>
>>
>> Moving to GitHub should be an excellent occasion to revisit how the CLA
>> works and provide better integration, e.g.: by using something like
>> CLAHub[1].
>>
>
> FYI we're looking at CLAHub as a possible solution for this (either
> directly or with a few modifications to tie it into our systems). No
> promises but it's on the table.
>
>  That's why we're proposing to ditch per-WG anything here. The way
>>> html-testsuite is set up, we already have subdirectories for html,
>>> canvas2d, and microdata. Those are all from the HTML WG, but they're
>>> just listed as the individual specs. We can keep on adding more specs in
>>> there (the Web Crypto people are looking to do that).
>>>
>>
>> That sounds good to me. It's the per WG siloing I'm opposed to, not the
>> one repository to rule them all idea.
>>
>
> Good! Well, it looks like everyone agrees... If we're forging ahead, I
> have admin rights to the repo so you know who to prod.
>
> --
> Robin Berjon - http://berjon.com/ - @robinberjon
>
>

Re: Proposal: moving tests to GitHub

2013-01-22 Thread Tobie Langel 
On 1/23/13 12:48 AM, "Julian Aubourg"  wrote:

>I love the idea of moving to github.
>The one-repo idea, while much simpler from a maintenance point of view,
>could easily be a burden on users that subscribe to it. Even more so for
>people who can merge PRs (and thus will receive an email for a PR
>initiated for any spec).
>
>Not saying it is blocking but it's something to keep in mind. Mail
>filters can go a long way here but filtering out specific specs kinda
>defeats the purpose of having so many eyes looking at everything.

It's also worth thinking about which solution will have more chances of
fostering a community of external contributors and reviewers. Strong but
very specialized contributors might not get noticed. Being the biggest
contributor to the XHR test suite might carry a lot more value than being
the 50th biggest contributor to W3C tests in general.

--tobie

Re: Proposal: moving tests to GitHub

2013-01-23 Thread Robin Berjon 

On 23/01/2013 00:48 , Julian Aubourg wrote:

The one-repo idea, while much simpler from a maintenance point of view,
could easily be a burden on users that subscribe to it. Even more so for
people who can merge PRs (and thus will receive an email for a PR
initiatedfor any spec).


It *could*. But we don't know that yet. Splitting is easy enough. So I 
reckon we can start with the simple, one-repo approach and if as it 
ramps up we find that produces too much volume in email (or any such 
thing that can be hard to manage) then we can cross the splitting 
bridge. One good thing is that the experiment might give us valuable 
information about what splitting lines make sense to our community. For 
instance, to take a random example, it might be that it makes sense to 
put all APIs together in one repo and all markup in another (I doubt 
that's the case, but it's just an example of a split that doesn't map to 
ours that could possibly emerge).


To put this more shortly: I'd rather only deal with the problems of 
actually getting a community now (for which a single point of rallying 
is helpful). I'll be overjoyed with having to deal with the problems 
that come with having built a successful community later.


And Tobie writed:

It's also worth thinking about which solution will have more chances of
fostering a community of external contributors and reviewers. Strong but
very specialized contributors might not get noticed. Being the biggest
contributor to the XHR test suite might carry a lot more value than being
the 50th biggest contributor to W3C tests in general.


This cuts both ways. Being the top contributor for a dozen smaller, less 
noticed APIs or features (e.g. Vibration, ruby markup) doesn't rate as 
high as being, say, 8th overall.


I certainly don't disagree that having a way of publicly recognising 
contributors (beyond peer recognition from those who track the PRs) 
would likely prove valuable. But again, I think that's something that we 
can shape as we go along. The requisite data is available through the 
API. You can extract overall contribution and you can filter it by root 
directories that it touched. I reckon we can get the same data 
irrespective of which approach we pick.


But, again, I'd rather we focused on getting it off the ground well and 
proper. When the gates get flooded we can reassess. At this point I 
should probably stop belabouring my point because I'm this close to 
using the word "agile".


--
Robin Berjon - http://berjon.com/ - @robinberjon

Re: Proposal: moving tests to GitHub

2013-01-23 Thread Florian Bösch 
I can't guess how important the whole "attribution" thing is. I can however
say that having a public repository on github makes it easier for
"drive-by" contributors to contribute something. The traditional process
captures almost none of these contributions. I can also tell that I (and
probably most drive-by contributors) aren't that interested in attribution,
we just want to have our tiny contribution jotted down at the right place
and then be done with it.


On Wed, Jan 23, 2013 at 12:46 PM, Robin Berjon  wrote:

> On 23/01/2013 00:48 , Julian Aubourg wrote:
>
>> The one-repo idea, while much simpler from a maintenance point of view,
>> could easily be a burden on users that subscribe to it. Even more so for
>> people who can merge PRs (and thus will receive an email for a PR
>> initiatedfor any spec).
>>
>
> It *could*. But we don't know that yet. Splitting is easy enough. So I
> reckon we can start with the simple, one-repo approach and if as it ramps
> up we find that produces too much volume in email (or any such thing that
> can be hard to manage) then we can cross the splitting bridge. One good
> thing is that the experiment might give us valuable information about what
> splitting lines make sense to our community. For instance, to take a random
> example, it might be that it makes sense to put all APIs together in one
> repo and all markup in another (I doubt that's the case, but it's just an
> example of a split that doesn't map to ours that could possibly emerge).
>
> To put this more shortly: I'd rather only deal with the problems of
> actually getting a community now (for which a single point of rallying is
> helpful). I'll be overjoyed with having to deal with the problems that come
> with having built a successful community later.
>
> And Tobie writed:
>
>  It's also worth thinking about which solution will have more chances of
>> fostering a community of external contributors and reviewers. Strong but
>> very specialized contributors might not get noticed. Being the biggest
>> contributor to the XHR test suite might carry a lot more value than being
>> the 50th biggest contributor to W3C tests in general.
>>
>
> This cuts both ways. Being the top contributor for a dozen smaller, less
> noticed APIs or features (e.g. Vibration, ruby markup) doesn't rate as high
> as being, say, 8th overall.
>
> I certainly don't disagree that having a way of publicly recognising
> contributors (beyond peer recognition from those who track the PRs) would
> likely prove valuable. But again, I think that's something that we can
> shape as we go along. The requisite data is available through the API. You
> can extract overall contribution and you can filter it by root directories
> that it touched. I reckon we can get the same data irrespective of which
> approach we pick.
>
> But, again, I'd rather we focused on getting it off the ground well and
> proper. When the gates get flooded we can reassess. At this point I should
> probably stop belabouring my point because I'm this close to using the word
> "agile".
>
>
> --
> Robin Berjon - http://berjon.com/ - @robinberjon
>
>

Re: Proposal: moving tests to GitHub

2013-01-23 Thread Arthur Barstow 

On 1/22/13 5:53 AM, ext Odin Hørthe Omdal wrote:
We just had a small discussion on webapps-testsuite [1] about the 
possibility of moving the webapps tests.  I was wrongly under the 
impression that we had discussed this before (hey, confusion is not a 
crime ;) ).  Now that HTML has done the move, I think it is time for 
us to look at it too.  Ms2ger and Robin, which did most of the HTML 
testsuite [2] move were happy to help [3].


Ms2ger proposed merging our repository with HTML at the same time and 
not necessarily having one repository for each group.  I was already 
thinking such a move might be beneficial to do for webapps and 
webappsec, but it might be even more simple to also have html 
testsuite in that merge.


Robin wrote an email describing what they did for the HTML move, some 
of which should be relevant for our potential move too [4]. Of 
particular interest the possibility for having tests in folders by 
section of the spec, and making submissions pull requests.



I'm interested in such a move because it'll make our tests more 
visible, and easier to contribute to.  Just this weekend the Server 
Sent Events spec got 6 pull requests from a GitHub user "Yaffle" [5].  
Note that that particular repository will be moved into however our 
new test repository structure will be, so it probably won't remain as 
a separate repository.


Before we start a CfC to change WebApps' agreed testing process 
[Testing], please make a clear proposal regarding the submission 
process, approval process, roles, etc. as is defined in [Testing] and 
its references. (My preference is for you to document the new process, 
expectations, etc. in WebApps' Public wiki, rooted at 
).


Also, what is the expectation regarding [Framework]? Does your proposal 
include still using it? If yes, will there be automagic mirroring of 
github changes to w3c-test.org?


-Thanks, AB

[Testing] 
[Framework] 






[1] 
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0019.html

[2] https://github.com/w3c/html-testsuite
[3] 
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0022.html
[4] 
http://lists.w3.org/Archives/Public/public-webapps-testsuite/2013Jan/0024.html

[5] https://github.com/w3c/event-source/pulls

Re: Proposal: moving tests to GitHub

2013-01-23 Thread Robin Berjon 

On 23/01/2013 13:01 , Arthur Barstow wrote:

Before we start a CfC to change WebApps' agreed testing process
[Testing], please make a clear proposal regarding the submission
process, approval process, roles, etc. as is defined in [Testing] and
its references. (My preference is for you to document the new process,
expectations, etc. in WebApps' Public wiki, rooted at
).


I'll leave that to Odin since he's been driving this, but I'll be happy 
to comment and help.



Also, what is the expectation regarding [Framework]? Does your proposal
include still using it? If yes, will there be automagic mirroring of
github changes to w3c-test.org?


We shouldn't mix using w3c-test.org and using the current framework. The 
former we definitely keep on using — we're right now working on the sync 
from our TS to that server.


The framework I think was a proof of concept but has endemic problems 
(one of which being that it is painful and costly to maintain). I know 
we can do much better. I'll bang my head with that of a number of other 
people and I'm sure we'll come up with something. This, however, is 
orthogonal to the GitHub reorg. You can keep using the existing 
framework after the GitHub move, and it would still have the same 
problems if we don't move.


--
Robin Berjon - http://berjon.com/ - @robinberjon

Re: Proposal: moving tests to GitHub

2013-01-23 Thread Odin Hørthe Omdal 

Arthur Barstow wrote:
Before we start a CfC to change WebApps' agreed testing process  
[Testing], please make a clear proposal regarding the submission  
process, approval process, roles, etc. as is defined in [Testing] and  
its references. (My preference is for you to document the new process,  
expectations, etc. in WebApps' Public wiki, rooted at  
).


Okay! I will try to have something we can look at and discuss tomorrow. :-)

--
Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com

Re: Proposal: moving tests to GitHub

2013-01-24 Thread Odin Hørthe Omdal 

Arthur Barstow wrote:
Before we start a CfC to change WebApps' agreed testing process  
[Testing], please make a clear proposal regarding the submission  
process, approval process, roles, etc. as is defined in [Testing] and  
its references. (My preference is for you to document the new process,  
expectations, etc. in WebApps' Public wiki, rooted at  
).


I've written (well, copied and changed) a document at:

http://www.w3.org/wiki/Webapps/Submitting_tests

It might not have everything required right now, but I think it's a good  
start. :-)


--
Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com

Re: Proposal: moving tests to GitHub

2013-01-25 Thread James Graham 

On 01/24/2013 07:22 PM, Odin Hørthe Omdal wrote:

Arthur Barstow wrote:

Before we start a CfC to change WebApps' agreed testing process
[Testing], please make a clear proposal regarding the submission
process, approval process, roles, etc. as is defined in [Testing] and
its references. (My preference is for you to document the new process,
expectations, etc. in WebApps' Public wiki, rooted at
).


I've written (well, copied and changed) a document at:

 http://www.w3.org/wiki/Webapps/Submitting_tests

It might not have everything required right now, but I think it's a good
start. :-)


FWIW that looks good to me. At risk of bikeshedding, I think that 
calling a repo with tests for non-HTML specs "html-testsuite" is 
confusing and will make the repository harder to find, especially since 
the people who are aware that html is not a catch-all term are also the 
people most likely to be writing tests. Some more generic name like 
"web-platform-testsuite" seems better.

Re: Proposal: moving tests to GitHub

2013-01-25 Thread Tobie Langel 
> FWIW that looks good to me. At risk of bikeshedding, I think that calling a 
> repo with tests for non-HTML specs "html-testsuite" is confusing and will 
> make the repository harder to find, especially since the people who are aware 
> that html is not a catch-all term are also the people most likely to be 
> writing tests. Some more generic name like "web-platform-testsuite" seems 
> better.

Couldn't agree more.

--tobie

Re: Proposal: moving tests to GitHub

2013-01-25 Thread Tobie Langel 
On Jan 24, 2013, at 1:24 PM, "Odin Hørthe Omdal"  wrote:

> Arthur Barstow wrote:
>> Before we start a CfC to change WebApps' agreed testing process [Testing], 
>> please make a clear proposal regarding the submission process, approval 
>> process, roles, etc. as is defined in [Testing] and its references. (My 
>> preference is for you to document the new process, expectations, etc. in 
>> WebApps' Public wiki, rooted at ).
> 
> I've written (well, copied and changed) a document at:
> 
>http://www.w3.org/wiki/Webapps/Submitting_tests
> 
> It might not have everything required right now, but I think it's a good 
> start. :-)

This is awesome. Thanks for writing it.

--tobie

Re: Proposal: moving tests to GitHub

2013-01-31 Thread Arthur Barstow 

On 1/24/13 1:22 PM, ext Odin Hørthe Omdal wrote:

Arthur Barstow wrote:
Before we start a CfC to change WebApps' agreed testing process 
[Testing], please make a clear proposal regarding the submission 
process, approval process, roles, etc. as is defined in [Testing] and 
its references. (My preference is for you to document the new 
process, expectations, etc. in WebApps' Public wiki, rooted at 
).


I've written (well, copied and changed) a document at:

http://www.w3.org/wiki/Webapps/Submitting_tests

It might not have everything required right now, but I think it's a 
good start. :-)



Thanks Odin (and sorry for the delayed reply)!

As I said during one of the testing breakouts in Lyon, ultimately I 
suspect the saying "beggars can't be choosy" will trump. However, AFAIK, 
currently, only one of WebApps' thirty active specs actually has an 
"outside" contribution. As such, and without any information about a 
relatively high probability we will get contributions from others, this 
move still seems like a lot of "make work".


Before a CfC is started, I would like to hear from Kris and/or PLH re 
how the move went for the HTMLWG. For instance, were there any some 
major "gotchas", were there any negative side-effects, etc. Kris, PLH - 
would you please provide a short summary of the move?


I agree with James and Tobie re the name and structure issue. James' 
proposal was fine or something a bit shorter like "webapi-tests". I also 
wonder if it would be useful to separate the DOM tests (e.g. 
"dom-tests") although I don't feel strongly on that.


Re section numbers - that seems like make work, especially for short-ish 
specs (e.g. Progress Events). I think using section numbers should be 
optional (and that metadata be included in the tests themselves). Are 
you actually proposing to add section numbers for every test suite that 
you copy?


What is the expectation for what I will characterize as "legacy" specs 
like Marcos' widget test suites? Marcos?


Test Facilitators and Test Submitters - if you haven't already provided 
input on Odin's proposal, please do so. My expectation is that Odin and 
the other volunteers will do all of the copying work.


-Thanks, AB

Re: Proposal: moving tests to GitHub

2013-01-31 Thread Odin Hørthe Omdal 
On Thu, 31 Jan 2013 18:13:15 +0100, Arthur Barstow   
wrote:
However, AFAIK, currently, only one of WebApps' thirty active specs  
actually has an "outside" contribution.


I should've already left work, so I'll just reply to this sentence quickly  
:-)


With that you mean Server Sent Events?  The /only/ webapps spec (that I  
know of) that actually is at GitHub and not at w3c dvcs?  In that case we  
have "outside" contributions on 100% of the tests on GitHub, and 0% on the  
tests at w3c dvcs.


Although, I don't really know how to count TestTWF, but there's quite a  
lot of webapps tests there as well.  Rebecca Hauck pushed them to dvcs a  
while ago, although that was a good time after the event.  I'll hazard to  
guess that if they could do the pull requests right there at the event  
(like what happened for Server Sent Events), we might have a higher retain  
rate.  Because I think it'd make people feel easier/more involved.   
Speculation, but not totally unfounded :-)



Many other good points. I guess Robin, James, Tobie or someone else will  
already have replied by tomorrow. :]

--
Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software, http://opera.com

Re: Proposal: moving tests to GitHub

2013-01-31 Thread Rebecca Hauck 
I guess I should chime in.

Yes I submitted a batch of tests from TestTWF some time after the Paris
event. After having a pretty bad experience with Mercurial earlier in the
year at TestTWF San Francisco, we made a conscious choice to eliminate it
in Paris and use DropBox instead. It was marginally better if only for the
fact that people were actually writing tests rather than futzing with
Mercurial for half the day (this happened at the SF event - it got very
ugly for some).

The main flaw in the DropBox process was getting the tests moved to the
respective repositories. It was work we recognized and agreed to do up
front in favor of getting more people writing tests.  We first reached out
to all the test authors individually, gave them the Mercurial info, and
offered assistance getting the tests submitted (hence the time lag). For
whatever reasons, very few actually took the steps to submit themselves.
Some were even under the impression that the tests would be submitted for
them, a reasonable assumption based on the W3C Grant of License they
signed at the event.  So, to not lose those tests, I ended up submitting
them on behalf of the authors.

Since I'm not in the webapps working group, I had to first get access to
the repository. I was told that that to get write access, I (probably) had
to join the working group [1]. So it's sort of by definition that there
are no "outsider" submissions in dvcs, no?  Either way, after several
emails with several people, after about 5 days, I got write access and
pushed. I'm a semi-insider and even my experience wasn't that great. And
yes Odin, I completely agree that direct submission through pull requests
at the event would definitely have been more engaging with newcomers. I'm
pretty sure this will be addressed correctly at a future event.

And for lack of a better tracking mechanism, I added a  comment to each tests and in some cases, put them in a
similarly named folder in the repo. If you grep the webapps test dir,
you'll find more than one. :)

-Rebecca

[1] 
http://lists.w3.org/Archives/Public/public-test-infra/2012OctDec/0025.html

On 1/31/13 9:47 AM, "Odin Hørthe Omdal"  wrote:

>On Thu, 31 Jan 2013 18:13:15 +0100, Arthur Barstow
>
>wrote:
>> However, AFAIK, currently, only one of WebApps' thirty active specs
>> actually has an "outside" contribution.
>
>I should've already left work, so I'll just reply to this sentence
>quickly  
>:-)
>
>With that you mean Server Sent Events?  The /only/ webapps spec (that I
>know of) that actually is at GitHub and not at w3c dvcs?  In that case we
> 
>have "outside" contributions on 100% of the tests on GitHub, and 0% on
>the  
>tests at w3c dvcs.
>
>Although, I don't really know how to count TestTWF, but there's quite a
>lot of webapps tests there as well.  Rebecca Hauck pushed them to dvcs a
>while ago, although that was a good time after the event.  I'll hazard to
> 
>guess that if they could do the pull requests right there at the event
>(like what happened for Server Sent Events), we might have a higher
>retain  
>rate.  Because I think it'd make people feel easier/more involved.
>Speculation, but not totally unfounded :-)
>
>
>Many other good points. I guess Robin, James, Tobie or someone else will
>already have replied by tomorrow. :]
>-- 
>Odin Hørthe Omdal (Velmont/odinho) · Core, Opera Software,
>http://opera.com
>

Re: Proposal: moving tests to GitHub

2013-01-31 Thread Tobie Langel 
On 1/31/13 9:13 AM, "Arthur Barstow"  wrote:

>As I said during one of the testing breakouts in Lyon, ultimately I
>suspect the saying "beggars can't be choosy" will trump. However, AFAIK,
>currently, only one of WebApps' thirty active specs actually has an
>"outside" contribution. As such, and without any information about a
>relatively high probability we will get contributions from others, this
>move still seems like a lot of "make work".

Ultimately, that's a chicken and egg problem. Moving to GitHub doesn't
guarantee external contributions (there are aspects beyond using Git(Hub)
to involve and retain outside contributors), but the current solution
clearly prevents those.

If crowd-sourcing is part of our strategy to get more tests, (and the
testing meeting we had this week seems to imply it is), then moving to
GitHub is a requirement.

--tobie

Re: Proposal: moving tests to GitHub

2013-02-01 Thread Arthur Barstow 

On 2/1/13 2:04 AM, ext Tobie Langel wrote:

On 1/31/13 9:13 AM, "Arthur Barstow"  wrote:


As I said during one of the testing breakouts in Lyon, ultimately I
suspect the saying "beggars can't be choosy" will trump. However, AFAIK,
currently, only one of WebApps' thirty active specs actually has an
"outside" contribution. As such, and without any information about a
relatively high probability we will get contributions from others, this
move still seems like a lot of "make work".

Ultimately, that's a chicken and egg problem. Moving to GitHub doesn't
guarantee external contributions (there are aspects beyond using Git(Hub)
to involve and retain outside contributors), but the current solution
clearly prevents those.


One of things I wondering about is - after you leave your Fellow 
position [BTW, that's totally wicked so congrats on that!], and Robin 
has moved on to `greener pastures` and Odin has moved on to be CEO of 
Opera - if/when there are problems with GH, who are we gonna' call? Hg, 
despite its shortcomings, is backed by the W3C's crack SysTeam. Do we 
get an equivalent service from GH?



If crowd-sourcing is part of our strategy to get more tests, (and the
testing meeting we had this week seems to imply it is), then moving to
GitHub is a requirement.


Yes, those are good points and I'm wondering if there really needs to be 
a binary choice here or if there could be advantages to using both. For 
example, set up a skeleton structure on GH and if/when tests are 
submitted, the Test Facilitator could review them and copy the `good 
ones` to Hg.


-AB

Re: Proposal: moving tests to GitHub

2013-02-01 Thread Arthur Barstow 

On 1/31/13 3:18 PM, ext Rebecca Hauck wrote:

Yes I submitted a batch of tests from TestTWF some time after the Paris
event.


OK, thanks for this clarification (I just noticed your submissions for 
IDB and DOMEvents). Sorry Odin for having missed those!



Since I'm not in the webapps working group, I had to first get access to
the repository. I was told that that to get write access, I (probably) had
to join the working group [1].


Yes, it certainly would have been easier if Adobe was a member of WebApps.

-AB

Re: Proposal: moving tests to GitHub

2013-02-01 Thread Tobie Langel 
On 2/1/13 5:52 AM, "Arthur Barstow"  wrote:
>On 1/31/13 3:18 PM, ext Rebecca Hauck wrote:
>> Since I'm not in the webapps working group, I had to first get access to
>> the repository. I was told that that to get write access, I (probably)
>>had
>> to join the working group [1].
>
>Yes, it certainly would have been easier if Adobe was a member of WebApps.

Think we all agree that given tests are completely unrelated to IP
concerns, there are no reasons to warrant WG membership in order to be
able to contribute them.

Any process around test contributions that works better for group members
than it does for non-members hinders our ability to obtain external
contributions. One of my goals during my fellowship is to find solutions
around these issues.

--tobie

Re: Proposal: moving tests to GitHub

2013-02-01 Thread Tobie Langel 
On 2/1/13 4:23 AM, "Arthur Barstow"  wrote:
>One of things I wondering about is - after you leave your Fellow
>position [BTW, that's totally wicked so congrats on that!], and Robin
>has moved on to `greener pastures` and Odin has moved on to be CEO of
>Opera - if/when there are problems with GH, who are we gonna' call? Hg,
>despite its shortcomings, is backed by the W3C's crack SysTeam. Do we
>get an equivalent service from GH?

This is a valid concern we need to address. We need to define the
benefits, risks and risk mitigation strategies of moving to GitHub and
decide whether or not this is something that makes sense. I'm going to
start a doc on this that I'll use across WGs and socialize here first.

I'd really like to see input on this from GitHub doubters so we can really
address their concerns.

>> If crowd-sourcing is part of our strategy to get more tests, (and the
>> testing meeting we had this week seems to imply it is), then moving to
>> GitHub is a requirement.
>
>Yes, those are good points and I'm wondering if there really needs to be
>a binary choice here or if there could be advantages to using both. For
>example, set up a skeleton structure on GH and if/when tests are
>submitted, the Test Facilitator could review them and copy the `good
>ones` to Hg.

I have very large doubts about strategies that involve hum a intervention
to sync resources across different versioning systems.

--tobie

Re: Proposal: moving tests to GitHub

2013-02-04 Thread James Graham 

On 02/02/2013 12:50 AM, Tobie Langel wrote:

On 2/1/13 4:23 AM, "Arthur Barstow"  wrote:

One of things I wondering about is - after you leave your Fellow
position [BTW, that's totally wicked so congrats on that!], and Robin
has moved on to `greener pastures` and Odin has moved on to be CEO of
Opera - if/when there are problems with GH, who are we gonna' call? Hg,
despite its shortcomings, is backed by the W3C's crack SysTeam. Do we
get an equivalent service from GH?


In the broader context, git[hub] is very much the low-risk alternative. 
Whilst there are certainly people using mercurial, git is far more 
popular. Git skills are far more common in the marketplace, and far more 
common amongst people who are likely to join working groups. In addition 
the tooling support for git is much better; even Microsoft are adding 
git support to their tools these days ;) These things have a very real 
impact on us. For example I spent some time searching for a code review 
tool we could use with hg and came up blank. With git[hub] we get a 
mediocre one for free and the possibility of using a very good one once 
I add github API support.



If crowd-sourcing is part of our strategy to get more tests, (and the
testing meeting we had this week seems to imply it is), then moving to
GitHub is a requirement.


Yes, those are good points and I'm wondering if there really needs to be
a binary choice here or if there could be advantages to using both. For
example, set up a skeleton structure on GH and if/when tests are
submitted, the Test Facilitator could review them and copy the `good
ones` to Hg.


I have very large doubts about strategies that involve hum a intervention
to sync resources across different versioning systems.


Yes. Having only a skeleton setup on github would mean that people using 
that would be unable to fix existing tests. If a file changed on github 
that had already been copied across to hg, it would be an enormous pain 
to work out whether it is safe to port the changes. I think you would 
have to use a write/copy/delete workflow, which would be confusing as 
people wouldn't be able to find their contributions.


Having fully repositories in both places is also fairly painful, but 
tolerable if the commits are in on place only (if you start allowing 
commits in both places, you can end up with a very difficult 
synchronisation problem).

Re: Proposal: moving tests to GitHub

2013-02-04 Thread Robin Berjon 

On 31/01/2013 18:13 , Arthur Barstow wrote:

As I said during one of the testing breakouts in Lyon, ultimately I
suspect the saying "beggars can't be choosy" will trump. However,
AFAIK, currently, only one of WebApps' thirty active specs actually
has an "outside" contribution. As such, and without any information
about a relatively high probability we will get contributions from
others, this move still seems like a lot of "make work".


Aside from the external contributions that others have already pointed 
out, I think it's worth putting this statement in some literary perspective.



"But Mr Dent, the plans have been available in the local planning office 
for the last nine months."


"Oh yes, well as soon as I heard I went straight round to see them, 
yesterday afternoon. You hadn't exactly gone out of your way to call 
attention to them, had you? I mean, like actually telling anybody or 
anything."


"But the plans were on display ..."

"On display? I eventually had to go down to the cellar to find them."

"That's the display department."

"With a flashlight."

"Ah, well the lights had probably gone."

"So had the stairs."

"But look, you found the notice didn't you?"

"Yes," said Arthur, "yes I did. It was on display in the bottom of a 
locked filing cabinet stuck in a disused lavatory with a sign on the 
door saying 'Beware of the Leopard'."


-- Hitchhiker's Guide to the Galaxy, Douglas Adams


Seriously: we've had our test suites locked up on an unknown server, in 
an obsolete version control system that's protected by credentials that 
are hard to get. Shockingly enough, we have seen *some* external 
contribution.


Additionally, we also have to take the productivity of existing 
contributors into account. Even if no external contributor shows up, 
switching to git will already save work from existing contributors fast 
enough that any work involved in transitioning will be made up for 
within weeks.



Before a CfC is started, I would like to hear from Kris and/or PLH re
 how the move went for the HTMLWG. For instance, were there any some
 major "gotchas", were there any negative side-effects, etc. Kris,
PLH - would you please provide a short summary of the move?


It went like a breeze. The major difficulty was the submissions backlog, 
but then it's better for it to be a problem than to just linger on as 
had been the case to date.


One thing to watch out for is that it seems to have been relatively 
common for tests in the "submissions" directory to be *copied* to 
"approved" without being removed from their original directory. I 
detected quite a lot of that going on.


So overall, zero negative effects, afar better workflow, and new 
contributors we'd never heard of before.



Re section numbers - that seems like make work, especially for
short-ish specs (e.g. Progress Events). I think using section numbers
should be optional (and that metadata be included in the tests
themselves). Are you actually proposing to add section numbers for
every test suite that you copy?


Section numbers don't fly, but using section IDs to produce a tree of 
tests works really well (and you can automate the creation of the 
initial tree trivially).


It's far better than metadata because metadata is copied, goes awry, 
etc. whereas a file's location tends to just be correct. It's metadata 
but with usability taken into account.



What is the expectation for what I will characterize as "legacy"
specs like Marcos' widget test suites? Marcos?


I would say: whoever wants to include their stuff can include it, so 
long as it's legit content related to a spec.


--
Robin Berjon - http://berjon.com/ - @robinberjon