Polynomial regression - Warning
Polynomial regression was where I came into numerical analysis at the beginning of my academic career 40 years ago. It is a dangerously ill-conditioned problem, meaning that the regression parameters are untrustworthy, though the "fit", i.e., the model, may be useful if the calculations are done properly. There are ways of doing it that are "less dangerous" using orthogonal polynomials. Just a warning that it should not be a high priority to add. Let the Excel users drive off the cliff. If it is added for Excel compatibility, then I'd still recommend a "Gnumeric does Vista" and pop up warnings asking if the dangerous move should be allowed. Come to think of it, one could borrow exactly the Vista popups so folk would blame Bill. Seriously, think if you really need polynomial regression or can use a less troublesome approach. JN ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
Re: Polynomial regression - Warning
Lovely repartee, just the sophisticated answer that gnumeric brings to the spreadsheet world. Any chance you can craft this into a good popup dialog? e.g. You are trying to use SOME_METHOD which exists in gnumeric only to allow compatibility with other older spreadsheet programs. The computations involved in that method are known to be exceedingly problematic. However, you might be able to solve your problem using SOME_OTHER_METHOD. Would you like to use that instead? Of course, this is all over my head. I'm merely hoping to leverage your 40 years of knowledge into a helpful dialog. --adrian On Tue, 2007-10-09 at 14:46 -0400, Prof J C Nash wrote: > Polynomial regression was where I came into numerical analysis at the > beginning of my academic career 40 years ago. It is a dangerously > ill-conditioned problem, meaning that the regression parameters are > untrustworthy, though the "fit", i.e., the model, may be useful if the > calculations are done properly. There are ways of doing it that are > "less dangerous" using orthogonal polynomials. > > Just a warning that it should not be a high priority to add. Let the > Excel users drive off the cliff. If it is added for Excel compatibility, > then I'd still recommend a "Gnumeric does Vista" and pop up warnings > asking if the dangerous move should be allowed. Come to think of it, one > could borrow exactly the Vista popups so folk would blame Bill. > > Seriously, think if you really need polynomial regression or can use a > less troublesome approach. > > JN > > ___ > gnumeric-list mailing list > gnumeric-list@gnome.org > http://mail.gnome.org/mailman/listinfo/gnumeric-list ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
Re: Polynomial regression - Warning
The type of dialog Adrian suggests would be a very sensible feature. Sometimes one really does need to do the polynomial regression, even if to show the issues, so it is not right to completely bar an approach. However, when we have an interested person, it's a good opportunity to point them towards potentially better methods. There was some discussion of such dialogs in the 80s for statistical software, and there's some of these ideas in some of the packages. Certainly I'm interested in helping out on this sort of thing, and have been gradually clawing away at other stuff I have to do so I can return to the test spreadsheets, which are one way to bring the issues to light. JN Adrian Custer wrote: > Lovely repartee, just the sophisticated answer that gnumeric brings to > the spreadsheet world. > > Any chance you can craft this into a good popup dialog? e.g. > > You are trying to use SOME_METHOD which exists in gnumeric only > to allow compatibility with other older spreadsheet programs. > The computations involved in that method are known to be > exceedingly problematic. However, you might be able to solve > your problem using SOME_OTHER_METHOD. Would you like to use that > instead? > > Of course, this is all over my head. I'm merely hoping to leverage your > 40 years of knowledge into a helpful dialog. > > --adrian > > > On Tue, 2007-10-09 at 14:46 -0400, Prof J C Nash wrote: > >> Polynomial regression was where I came into numerical analysis at the >> beginning of my academic career 40 years ago. It is a dangerously >> ill-conditioned problem, meaning that the regression parameters are >> untrustworthy, though the "fit", i.e., the model, may be useful if the >> calculations are done properly. There are ways of doing it that are >> "less dangerous" using orthogonal polynomials. >> >> Just a warning that it should not be a high priority to add. Let the >> Excel users drive off the cliff. If it is added for Excel compatibility, >> then I'd still recommend a "Gnumeric does Vista" and pop up warnings >> asking if the dangerous move should be allowed. Come to think of it, one >> could borrow exactly the Vista popups so folk would blame Bill. >> >> Seriously, think if you really need polynomial regression or can use a >> less troublesome approach. >> >> JN >> >> ___ >> gnumeric-list mailing list >> gnumeric-list@gnome.org >> http://mail.gnome.org/mailman/listinfo/gnumeric-list >> ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
Re: Polynomial regression - Warning
Unfortunately we cannot really know if the user's use of the tool is sane. Yes, a chain saw can be dangerous but it is fundamentally a good tool when used right. What we can do -- at least for the GUI regression tool -- is to check the condition number of the relevant matrix. We do that already and warn when the result starts to lose significance. And err when it is completely gone. Morten ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
Re: Polynomial regression - Warning
Use of the condition number is sensible. If one can detect polynomial regression, however, there are alternatives that may improve the condition, and it would be nice to tell folk. That's not necessarily Gnumeric's job, of course. And there may be more important "bad habits" that should be dealt with first. My experience is that users of polynomial regression don't want to know they are doing something silly, and are wilfully blind to the issues. There's of course the annoying problem that condition numbers are not easy to estimate Another story. I'll hopefully be able to provide some input and reasonable (i.e., plug out, plug in) fixes after a few more months of wading through other projects. JN Morten Welinder wrote: > Unfortunately we cannot really know if the user's use of the tool is sane. > Yes, a chain saw can be dangerous but it is fundamentally a good tool > when used right. > > What we can do -- at least for the GUI regression tool -- is to check the > condition number of the relevant matrix. We do that already and warn > when the result starts to lose significance. And err when it is completely > gone. > > Morten > ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
Re: Polynomial regression - Warning
Morten Welinder wrote: > Yes, a chain saw can be dangerous but it is fundamentally a good tool > when used right. > My mother's uncle, who had been totally blind since the age of two, actually bought a chainsaw to use for cutting firewood. He only lost one of his fingers. ___ gnumeric-list mailing list gnumeric-list@gnome.org http://mail.gnome.org/mailman/listinfo/gnumeric-list
