[Rd] cor.test(method = spearman, exact = TRUE) not exact (PR#14095)

2009-11-30 Thread dsimcha 
Full_Name: David Simcha
Version: 2.10
OS: Windows XP Home
Submission from: (NULL) (173.3.208.5)


 a - c(1:10)
 b - c(1:10)
 cor.test(a, b, method = spearman, alternative = greater, exact = TRUE)

Spearman's rank correlation rho

data:  a and b 
S = 0, p-value  2.2e-16
alternative hypothesis: true rho is greater than 0 
sample estimates:
rho 
  1 

 1 / factorial(10)
[1] 2.755732e-07

Since we have perfect rank correlation and only one permutation out of 10! could
give this for N = 10, the p-value should be 1/10!.  Reading the code in prho.c,
it appears that the exact calculation uses the Edgeworth approximation for N 
9.  This makes sense because, for similar examples with N = 9, the results are
as expected (1 / N!).

The exact p-value calculation is good enough for most practical purposes, but
is clearly not exact.  Some informal testing I've done indicates that it can
even be less accurate than the approximate p-value calculation in some cases. 
I think it's absurd to call these p-values exact when they are clearly based
on an asymptotic approximation that can be off by orders of magnitude in some
cases.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] cor.test(method = spearman, exact = TRUE) not exact (PR#14095)

2009-11-30 Thread Petr Savicky 
On Mon, Nov 30, 2009 at 04:00:12AM +0100, dsim...@gmail.com wrote:
  a - c(1:10)
  b - c(1:10)
  cor.test(a, b, method = spearman, alternative = greater, exact = TRUE)
 
 Spearman's rank correlation rho
 
 data:  a and b 
 S = 0, p-value  2.2e-16
 alternative hypothesis: true rho is greater than 0 
 sample estimates:
 rho 
   1 
 
  1 / factorial(10)
 [1] 2.755732e-07
 
 Since we have perfect rank correlation and only one permutation out of 10! 
 could
 give this for N = 10, the p-value should be 1/10!.  Reading the code in 
 prho.c,
 it appears that the exact calculation uses the Edgeworth approximation for 
 N 
 9.  This makes sense because, for similar examples with N = 9, the results 
 are
 as expected (1 / N!).

The Edgeworth approximation is not exact, although the error is quite small.
Computing the exact values has time complexity roughly 2^n, if Ryser's formula
for permanent is used. In cases, where one needs the exact values for small n,
it is possible to use the package pspearman at CRAN, which includes precomuted
values up to n=22. See also the paper
  M.A. van de Wiel and A. Di Bucchianico,
  Fast computation of the exact null distribution of Spearman's rho and Page's 
L statistic
  for samples with and without ties, J. Stat. Plann. Inf. 92 (2001), pp. 
133-145.
which deals with this question.

  library(pspearman)
  a - c(1:10)
  b - c(1:10)
  out - spearman.test(a, b, alternative = greater, approximation=exact)
  out
  # Spearman's rank correlation rho
  # data:  a and b 
  # S = 0, p-value = 2.756e-07
  # alternative hypothesis: true rho is greater than 0 
  # sample estimates:
  # rho 
  #   1 
  out$p.value - 1/factorial(10) # [1] 0

PS.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel