The question really is what are some tried approaches to figure out how to measure the quality of a set of algorithms currently being used for clustering/classification?
And in thinking about this some more we also need to be able to regenerate models as soon as the business users tweak the weights associated with features inside a feature vector, we need to figure out a way to efficiently tie this into our online workflow which could show updated recommendations every few hours? When I say picking an algorithm on the fly what I mean is that we need to continuously test our basket of algorithms based on a new set of training data and make the determination offline as to which of the algorithms to use at that moment to regenerate our recommendations. > Date: Tue, 10 Apr 2012 14:08:17 -0500 > Subject: Re: Evalutation of recommenders > From: sro...@gmail.com > To: user@mahout.apache.org > > Picking an algorithm 'on the fly' is almost surely not realistic -- > well, I am not sure what eval process you would run in milliseconds. > But it's also unnecessary; you usually run evaluations offline on > training/test data that reflects real input, and then, the resulting > tuning should be fine for that real input that comes the next day. > > Is that really the question, or are you just asking about how you > measure the quality of clustering or a classifier? > > On Tue, Apr 10, 2012 at 10:41 AM, Saikat Kanjilal <sxk1...@hotmail.com> wrote: > > > > Hi everyone,We're looking at building out some clustering and > > classification algorithms using mahout and one of the things we're also > > looking at doing is to build performance metrics around each of these > > algorithms, as we go down the path of choosing the best model in an > > iterative closed feedback loop (i.e. our business users manipulate weights > > for each attribute for our feature vectors->we use these changes to > > regenerate an asynchronous model using the appropriate > > clustering/classification algorithms and then replenish our online > > component with this newly recalculated data for fresh recommendations). > > So our end goal is to have a basket of algorithms and use a set of > > performance metrics to pick and choose the right algorithm on the fly. I > > was wondering if anyone has done this type of analysis before and if so are > > there approaches that have worked well and approaches that haven't when it > > comes to measuring the "quality" of each of the recommendation algorithms. > > Regards