Hi Frank > This is self-evident; what is not obvious is why the target function should be having the final word. Wasn't the word "over-refinement" introduced to describe exactly this: that the target function was wrong?
I assumed people were confusing 'over-refinement' with 'over-fitting'; there is no such thing as over-refinement (you won't find 'over-optimisation' mentioned in any textbooks on optimisation!), since by definition refining beyond convergence is a waste of time: it cannot result in any further significant changes in the parameters. Refinement is simply a method of solving a set of equations & clearly we want to solve those equations as accurately as possible. If we could obtain an exact solution without iteration we would use that; the mechanics of refinement and the path that the program by dint of numerical quirks happens to take to reach the solution are irrelevant as far as the user is concerned. Stopping the refinement before convergence will not produce more accurate parameters, it will just introduce random and unquantifiable errors. > Isn't this the purpose of cross-validation, to use an independent measure to judge when the refinement is *not* producing the "best" model? If the value of your chosen X-validation metric at convergence indicates a problem with the model, parameterisation, weighting etc then clearly the target function is not indeed the final word: the solution is to fix whatever is wrong and do the refinement again, until you get a satisfactory value for your metric. > > This may be true; but as it is independent of refinement, is it not > nevertheless the only measure I should trust? > > No there several possible functions of the test set (e.g. Hamilton Rfree, LLfree) that you could use, all potentially equally valid X-validation metrics. I would have more faith in a function such as LLfree in which the contributions of the reflections are at least weighted according to their reliability. It just seems bizarre that important decisions are being based on measurements that may have gross errors without taking those errors into account. > Or maybe what you intended to say: only trust refinements for which Rfree > decreases monotonically, because only then do you have a valid choice of > parameters. > No, as I indicated above, what Rfree does before convergence is attained is totally meaningless, only the value obtained _at_ convergence is meaningful as a X-validation statistic. We wouldn't be having this discussion if the refinement program omitted the meaningless intermediate values and only printed out the final Rfree or LLfree. I'm saying that Rfree is not the best X-validation metric because poorly measured data are not properly weighted: this is what Acta paper I referenced is saying. Cheers -- Ian
