CrossValidator splits the data into k sets, and then trains k times, holding out one subset for cross-validation each time. You are correct that you should actually withhold an additional test set, before you use CrossValidator, in order to get an unbiased estimate of the best model's performance.
On Tue, Nov 1, 2016 at 12:10 PM Nirav Patel <npa...@xactlycorp.com> wrote: > I am running classification model. with normal training-test split I can > check model accuracy and F1 score using MulticlassClassificationEvaluator. > How can I do this with CrossValidation approach? > Afaik, you Fit entire sample data in CrossValidator as you don't want to > leave out any observation from either testing or training. But by doing so > I don't have anymore unseen data on which I can run finalized model on. So > is there a way I can get Accuracy and F1 score of a best model resulted > from cross validation? > Or should I still split sample data in to training and test before running > cross validation against only training data? so later I can test it against > test data. > > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation>
