The CCO algorithm now supports a couple ways to limit indicators by “quality". The new way is by the value of LLR. We built a t-digest mechanism to look at the overall density produced with different thresholds. The higher the threshold, the lower the number of indicators and the lower the density of the resulting indicator matrix but also the higher the MAP score (of the full recommender). So MAP seems to increase monotonically until it breaks down.
This didn’t match my understanding of LLR, which is actually a test for non-correlation. I was expecting high scores to mean highly likelihood of non-correlation. So the actual formulation of the code must be reversing that so the higher the score the higher the likelihood that non-correlation is *false* (this is a treated as evidence of correlation) The next observation is that with high thresholds we get higher MAP scores from the recommender (expected) but this increases monotonically until it breaks down because there are so few indicators left. This leads us to the conclusion that MAP is not a good way to set the threshold. We tried to looking are precision (MAP) vs recall (number of people who get recs) and this gave ambiguous results with the data we had. Given my questions about how LLR is actually formulated in Mahout I’m unsure how to convert it into something like a confidence score or some other way to judge the threshold that would lead to good way to choose a threshold. Any ideas or illumination about how it’s being calculated or how to judge the threshold? Long description of motivation: LLR thresholds are needed when comparing conversion events to things that have very small dimensionality so maxIndicatorsPerIItem does not work well. For example a location by state where there are 50, maxIndicatorsPerItem defaults to 50 so you may end up with 50 very week indicators. If there are strong indicators in the data, thresholds should be the way to find them. This might lead to a few per item if the data supports it and this should then be useful. The question above is how to choose a threshold.
