Hi CW, It's not about the concept of the black box, none of the algorithms in sklearn are a blackbox. The question is about model validity. Is linear regression a valid representation of your data? That's what the train/test answers. You may think so, but only this process will answer it properly.
Matthieu Le mer. 5 juin 2019 à 01:46, C W <tmrs...@gmail.com> a écrit : > Thank you all for the replies. > > I agree that prediction accuracy is great for evaluating black-box ML > models. Especially advanced models like neural networks, or not-so-black > models like LASSO, because they are NP-hard to solve. > > Linear regression is not a black-box. I view prediction accuracy as an > overkill on interpretable models. Especially when you can use R-squared, > coefficient significance, etc. > > Prediction accuracy also does not tell you which feature is important. > > What do you guys think? Thank you! > > . > > On Mon, Jun 3, 2019 at 11:43 AM Andreas Mueller <t3k...@gmail.com> wrote: > >> This classical paper on statistical practices (Breiman's "two cultures") >> might be helpful to understand the different viewpoints: >> >> https://projecteuclid.org/euclid.ss/1009213726 >> >> >> On 6/3/19 12:19 AM, Brown J.B. via scikit-learn wrote: >> >> As far as I understand: Holding out a test set is recommended if you >>> aren't entirely sure that the assumptions of the model are held (gaussian >>> error on a linear fit; independent and identically distributed samples). >>> The model evaluation approach in predictive ML, using held-out data, relies >>> only on the weaker assumption that the metric you have chosen, when applied >>> to the test set you have held out, forms a reasonable measure of >>> generalised / real-world performance. (Of course this too is often not held >>> in practice, but it is the primary assumption, in my opinion, that ML >>> practitioners need to be careful of.) >>> >> >> Dear CW, >> As Joel as said, holding out a test set will help you evaluate the >> validity of model assumptions, and his last point (reasonable measure of >> generalised performance) is absolutely essential for understanding the >> capabilities and limitations of ML. >> >> To add to your checklist of interpreting ML papers properly, be cautious >> when interpreting reports of high performance when using 5/10-fold or >> Leave-One-Out cross-validation on large datasets, where "large" depends on >> the nature of the problem setting. >> Results are also highly dependent on the distributions of the underlying >> independent variables (e.g., 60000 datapoints all with near-identical >> distributions may yield phenomenal performance in cross validation and be >> almost non-predictive in truly unknown/prospective situations). >> Even at 500 datapoints, if independent variable distributions look >> similar (with similar endpoints), then when each model is trained on 80% of >> that data, the remaining 20% will certainly be predictable, and repeating >> that five times will yield statistics that seem impressive. >> >> So, again, while problem context completely dictates ML experiment >> design, metric selection, and interpretation of outcome, my personal rule >> of thumb is to do no-more than 2-fold cross-validation (50% train, 50% >> predict) when having 100+ datapoints. >> Even more extreme, using try 33% for training and 66% for validation (or >> even 20/80). >> If your model still reports good statistics, then you can believe that >> the patterns in the training data extrapolate well to the ones in the >> external validation data. >> >> Hope this helps, >> J.B. >> >> >> >> >> _______________________________________________ >> scikit-learn mailing >> listscikit-learn@python.orghttps://mail.python.org/mailman/listinfo/scikit-learn >> >> _______________________________________________ >> scikit-learn mailing list >> scikit-learn@python.org >> https://mail.python.org/mailman/listinfo/scikit-learn >> > _______________________________________________ > scikit-learn mailing list > scikit-learn@python.org > https://mail.python.org/mailman/listinfo/scikit-learn > -- Quantitative researcher, Ph.D. Blog: http://blog.audio-tk.com/ LinkedIn: http://www.linkedin.com/in/matthieubrucher
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
