I'm not convinced that swapping the labels of entire runs at a time
would constitute a stricter/more accurate test for overall accuracy than
swapping labels within a run, even in cases where it's possible (many
runs, different orderings of events within each run and the same number
of events in each run).
I can see that the procedure might be useful for some designs, such as
if there's concern about subtle order effects. But then the
whole-run-label-swapping would be used to check for order effects, not
the general accuracy.
I think the two label permutation methods (of trial labels or of entire
runs of labels) are estimating different null distributions. When it's
not possible to calculate all the label permutations (because there's
too many) we pick some permutations at random, hoping to have enough to
estimate the shape of the distribution if we were to have calculated
them all. But that's only ok if we choose the labels at random. Swapping
the labels of entire runs estimates a different distribution; a much
smaller number of possible label rearrangements.
I can see the logic, in that trials from different runs should be more
"independent" than trials from within the same run. But even trials from
different runs are not fully independent: they are from the same person,
there is a temporal order to the runs (so the person might be getting
tired as the experiment goes on), etc.
Jo
On 5/20/2011 10:02 AM, Yaroslav Halchenko wrote:
well, it is a number of combinations but they might be "illegal".
Non-parametric permutation testing requires the permuted units to be
independent. If you believe/assume/guarantee_somehow that your
samples are independent and free from order/run effects -- then go
ahead. If not -- you might take the permutation unit where you can
guarantee independence and that was my intent for the suggested
permutation of label sequences across runs.
Isn't what you're describing (run-label copying) a special case of
permuting the labels within each run?
yes in the sense that indeed by chance for some permutations you
might get an order which would match some run's order, but it would
not be very probable (depending on number of trials in a run of
cause).
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
