Just to be sure what you are trying to do:
A) compare the relevance of different similarities? this is something
the benchmark.quality package (actually pretty much unrelated from the
rest of the benchmark package!) does, if you have some e.g. TREC
collection or whatever to test with.
B) compare
I'd like to compare the relevance scores that are returned when using the
Similarity classes that are available in Lucene 4.x, and it seems like
using the Benchmark component would be a good way to do that. It looks like
the isn't currently a way to specify a Similarity class to use in the
config s