This is nothing whatsoever to do with by(), and it is cor, not Cor.
Try
X - cbind(NA, 1:3)
cor(X, use = complete)
cor(X, use = complete, method=spearman)
In short, cor() behaves differently when given a vector of NAs. That's
perfectly reasonable, as the ranks are undefined.
Since
I posted this on R help, and a few others responded indicating they too
were able to replicate the error as a function of missing data. I
believe this should not be the case and hence and reporting it here.
### Code provided on R-Help by Ivar Herfindal
# Simulate data
testdata -