Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
On 12/01/2011 08:00 PM, Ben quant wrote: The data I am using is the last file called l_yx.RData at this link (the second file contains the plots from earlier): http://scientia.crescat.net/static/ben/ The logistic regression model you are fitting assumes a linear relationship between x and the log odds of y; that does not seem to be the case for your data. To illustrate: x <- l_yx[,"x"] y <- l_yx[,"y"] ind1 <- x <= .002 ind2 <- (x > .002 & x <= .0065) ind3 <- (x > .0065 & x <= .13) ind4 <- (x > .0065 & x <= .13) > summary(glm(y[ind1]~x[ind1],family=binomial)) ... Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -2.791740.02633 -106.03 <2e-16 *** x[ind1] 354.98852 22.78190 15.58 <2e-16 *** > summary(glm(y[ind2]~x[ind2],family=binomial)) Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -2.158050.02966 -72.766 <2e-16 *** x[ind2] -59.929346.51650 -9.197 <2e-16 *** > summary(glm(y[ind3]~x[ind3],family=binomial)) ... Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -2.367206 0.007781 -304.22 <2e-16 *** x[ind3] 18.104314 0.346562 52.24 <2e-16 *** > summary(glm(y[ind4]~x[ind4],family=binomial)) ... Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -1.315110.08549 -15.383 <2e-16 *** x[ind4] 0.062610.08784 0.7130.476 To summarize, the relationship between x and the log odds of y appears to vary dramatically in both magnitude and direction depending on which interval of x's range we're looking at. Trying to summarize this complicated pattern with a single line is leading to the fitted probabilities near 0 and 1 you are observing (note that only 0.1% of the data is in region 4 above, although region 4 accounts for 99.1% of the range of x). -- Patrick Breheny Assistant Professor Department of Biostatistics Department of Statistics University of Kentucky __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
Thank you so much for your help.
The data I am using is the last file called l_yx.RData at this link (the
second file contains the plots from earlier):
http://scientia.crescat.net/static/ben/
Seems like the warning went away with pmin(x,1) but now the OR is over
15k. If I multiple my x's by 1000 I get a much more realistic OR. So I
guess this brings me to a much different question: aren't OR's comparable
between factors/data? In this case they don't seem to be. However, with
different data the OR's only change a very small amount (+8.0e-4) when I
multiply the x's by 1000. I don't understand.
Anyways, here is a run with the raw data and a run with your suggestion
(pmin(x,1)) that removed the error:
> l_logit = glm(y~x, data=as.data.frame(l_yx),
family=binomial(link="logit"))
> l_logit
Call: glm(formula = y ~ x, family = binomial(link = "logit"), data =
as.data.frame(l_yx))
Coefficients:
(Intercept)x
-2.2938.059
Degrees of Freedom: 690302 Total (i.e. Null); 690301 Residual
Null Deviance: 448800
Residual Deviance: 447100 AIC: 447100
> l_exp_coef = exp(l_logit$coefficients)[2]
> l_exp_coef
x
3161.781
> dim(l_yx)
[1] 690303 2
> l_yx = cbind(l_yx[,1],pmin(l_yx[,2],1))
> dim(l_yx)
[1] 690303 2
> colnames(l_yx) = c('y','x')
> mean(l_yx[,2])
[1] 0.01117248
> range(l_yx[,2])
[1] 0 1
> head(l_yx[,2])
[1] 0.00302316 0.07932130 0. 0.01779657 0.16083735 0.
> unique(l_yx[,1])
[1] 0 1
> l_logit = glm(y~x, data=as.data.frame(l_yx),
family=binomial(link="logit"))
> l_logit
Call: glm(formula = y ~ x, family = binomial(link = "logit"), data =
as.data.frame(l_yx))
Coefficients:
(Intercept)x
-2.3129.662
Degrees of Freedom: 690302 Total (i.e. Null); 690301 Residual
Null Deviance: 448800
Residual Deviance: 446800 AIC: 446800
> l_exp_coef = exp(l_logit$coefficients)[2]
> l_exp_coef
x
15709.52
Thanks,
Ben
On Thu, Dec 1, 2011 at 4:32 PM, peter dalgaard wrote:
>
> On Dec 1, 2011, at 23:43 , Ben quant wrote:
>
> > I'm not proposing this as a permanent solution, just investigating the
> warning. I zeroed out the three outliers and received no warning. Can
> someone tell me why I am getting no warning now?
>
> It's easier to explain why you got the warning before. If the OR for a one
> unit change is 3000, the OR for a 14 unit change is on the order of 10^48
> and that causes over/underflow in the conversion to probabilities.
>
> I'm still baffled at how you can get that model fitted to your data,
> though. One thing is that you can have situations where there are fitted
> probabilities of one corresponding to data that are all one and/or fitted
> zeros where data are zero, but you seem to have cases where you have both
> zeros and ones at both ends of the range of x. Fitting a zero to a one or
> vice versa would make the likelihood zero, so you'd expect that the
> algorithm would find a better set of parameters rather quickly. Perhaps the
> extremely large number of observations that you have has something to do
> with it?
>
> You'll get the warning if the fitted zeros or ones occur at any point of
> the iterative procedure. Maybe it isn't actually true for the final model,
> but that wouldn't seem consistent with the OR that you cited.
>
> Anyways, your real problem lies with the distribution of the x values. I'd
> want to try transforming it to something more sane. Taking logarithms is
> the obvious idea, but you'd need to find out what to do about the zeros --
> perhaps log(x + 1e-4) ? Or maybe just cut the outliers down to size with
> pmin(x,1).
>
> >
> > I did this 3 times to get rid of the 3 outliers:
> > mx_dims = arrayInd(which.max(l_yx), dim(l_yx))
> > l_yx[mx_dims] = 0
> >
> > Now this does not produce an warning:
> > l_logit = glm(y~x, data=as.data.frame(l_yx),
> family=binomial(link="logit"))
> >
> > Can someone tell me why occurred?
> >
> > Also, again, here are the screen shots of my data that I tried to send
> earlier (two screen shots, two pages):
> > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf
> >
> > Thank you for your help,
> >
> > Ben
> >
> > On Thu, Dec 1, 2011 at 3:25 PM, Ben quant wrote:
> > Oops! Please ignore my last post. I mistakenly gave you different data I
> was testing with. This is the correct data:
> >
> > Here you go:
> >
> > > attach(as.data.frame(l_yx))
> > > range(x[y==0])
> > [1] 0.0 14.66518
> > > range(x[y==1])
> > [1] 0.0 13.49791
> >
> >
> > How do I know what is acceptable?
> >
> > Also, here are the screen shots of my data that I tried to send earlier
> (two screen shots, two pages):
> > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf
> >
> > Thank you,
> >
> > Ben
> >
> > On Thu, Dec 1, 2011 at 3:07 PM, Ben quant wrote:
> > Here you go:
> >
> > > attach(as.data.frame(l_yx))
> > > range(x[y==1])
> > [1] -22500.746.
> > > range(x[y==0])
> > [1] -10076.5303653.0228
> >
> > How
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
On Dec 1, 2011, at 23:43 , Ben quant wrote: > I'm not proposing this as a permanent solution, just investigating the > warning. I zeroed out the three outliers and received no warning. Can someone > tell me why I am getting no warning now? It's easier to explain why you got the warning before. If the OR for a one unit change is 3000, the OR for a 14 unit change is on the order of 10^48 and that causes over/underflow in the conversion to probabilities. I'm still baffled at how you can get that model fitted to your data, though. One thing is that you can have situations where there are fitted probabilities of one corresponding to data that are all one and/or fitted zeros where data are zero, but you seem to have cases where you have both zeros and ones at both ends of the range of x. Fitting a zero to a one or vice versa would make the likelihood zero, so you'd expect that the algorithm would find a better set of parameters rather quickly. Perhaps the extremely large number of observations that you have has something to do with it? You'll get the warning if the fitted zeros or ones occur at any point of the iterative procedure. Maybe it isn't actually true for the final model, but that wouldn't seem consistent with the OR that you cited. Anyways, your real problem lies with the distribution of the x values. I'd want to try transforming it to something more sane. Taking logarithms is the obvious idea, but you'd need to find out what to do about the zeros -- perhaps log(x + 1e-4) ? Or maybe just cut the outliers down to size with pmin(x,1). > > I did this 3 times to get rid of the 3 outliers: > mx_dims = arrayInd(which.max(l_yx), dim(l_yx)) > l_yx[mx_dims] = 0 > > Now this does not produce an warning: > l_logit = glm(y~x, data=as.data.frame(l_yx), family=binomial(link="logit")) > > Can someone tell me why occurred? > > Also, again, here are the screen shots of my data that I tried to send > earlier (two screen shots, two pages): > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf > > Thank you for your help, > > Ben > > On Thu, Dec 1, 2011 at 3:25 PM, Ben quant wrote: > Oops! Please ignore my last post. I mistakenly gave you different data I was > testing with. This is the correct data: > > Here you go: > > > attach(as.data.frame(l_yx)) > > range(x[y==0]) > [1] 0.0 14.66518 > > range(x[y==1]) > [1] 0.0 13.49791 > > > How do I know what is acceptable? > > Also, here are the screen shots of my data that I tried to send earlier (two > screen shots, two pages): > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf > > Thank you, > > Ben > > On Thu, Dec 1, 2011 at 3:07 PM, Ben quant wrote: > Here you go: > > > attach(as.data.frame(l_yx)) > > range(x[y==1]) > [1] -22500.746. > > range(x[y==0]) > [1] -10076.5303653.0228 > > How do I know what is acceptable? > > Also, here are the screen shots of my data that I tried to send earlier (two > screen shots, two pages): > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf > > Thank you, > > Ben > > > On Thu, Dec 1, 2011 at 2:24 PM, peter dalgaard wrote: > > On Dec 1, 2011, at 21:32 , Ben quant wrote: > > > Thank you for the feedback, but my data looks fine to me. Please tell me if > > I'm not understanding. > > Hum, then maybe it really is a case of a transition region being short > relative to the range of your data. Notice that the warning is just that: a > warning. I do notice that the distribution of your x values is rather extreme > -- you stated a range of 0--14 and a mean of 0.01. And after all, an odds > ratio of 3000 per unit is only a tad over a doubling per 0.1 units. > > Have a look at range(x[y==0]) and range(x[y==1]). > > > > > > I followed your instructions and here is a sample of the first 500 values : > > (info on 'd' is below that) > > > > > d <- as.data.frame(l_yx) > > > x = with(d, y[order(x)]) > > > x[1:500] # I have 1's and 0's dispersed throughout > > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 > > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 > > 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 > > 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 > > [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 > > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 > > 1 0 0 0 0 0 0
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
I'm not proposing this as a permanent solution, just investigating the warning. I zeroed out the three outliers and received no warning. Can someone tell me why I am getting no warning now? I did this 3 times to get rid of the 3 outliers: mx_dims = arrayInd(which.max(l_yx), dim(l_yx)) l_yx[mx_dims] = 0 Now this does not produce an warning: l_logit = glm(y~x, data=as.data.frame(l_yx), family=binomial(link="logit")) Can someone tell me why occurred? Also, again, here are the screen shots of my data that I tried to send earlier (two screen shots, two pages): http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf Thank you for your help, Ben On Thu, Dec 1, 2011 at 3:25 PM, Ben quant wrote: > Oops! Please ignore my last post. I mistakenly gave you different data I > was testing with. This is the correct data: > > Here you go: > > > attach(as.data.frame(l_yx)) > > range(x[y==0]) > [1] 0.0 14.66518 > > range(x[y==1]) > [1] 0.0 13.49791 > > > How do I know what is acceptable? > > Also, here are the screen shots of my data that I tried to send earlier > (two screen shots, two pages): > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf > > Thank you, > > Ben > > On Thu, Dec 1, 2011 at 3:07 PM, Ben quant wrote: > >> Here you go: >> >> > attach(as.data.frame(l_yx)) >> > range(x[y==1]) >> [1] -22500.746. >> > range(x[y==0]) >> [1] -10076.5303653.0228 >> >> How do I know what is acceptable? >> >> Also, here are the screen shots of my data that I tried to send earlier >> (two screen shots, two pages): >> http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf >> >> Thank you, >> >> Ben >> >> >> On Thu, Dec 1, 2011 at 2:24 PM, peter dalgaard wrote: >> >>> >>> On Dec 1, 2011, at 21:32 , Ben quant wrote: >>> >>> > Thank you for the feedback, but my data looks fine to me. Please tell >>> me if I'm not understanding. >>> >>> Hum, then maybe it really is a case of a transition region being short >>> relative to the range of your data. Notice that the warning is just that: a >>> warning. I do notice that the distribution of your x values is rather >>> extreme -- you stated a range of 0--14 and a mean of 0.01. And after all, >>> an odds ratio of 3000 per unit is only a tad over a doubling per 0.1 units. >>> >>> Have a look at range(x[y==0]) and range(x[y==1]). >>> >>> >>> > >>> > I followed your instructions and here is a sample of the first 500 >>> values : (info on 'd' is below that) >>> > >>> > > d <- as.data.frame(l_yx) >>> > > x = with(d, y[order(x)]) >>> > > x[1:500] # I have 1's and 0's dispersed throughout >>> > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 >>> > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 >>> 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 >>> 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 >>> > [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 >>> 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >>> > >>> > # I get the warning still >>> > > l_df = as.data.frame(l_yx) >>> > > l_logit = glm(y~x, data=l_df, family=binomial(link="logit")) >>> > >>> > Warning message: >>> > glm.fit: fitted probabilities numerically 0 or 1 occurred >>> > >>> > # some info on 'd' above: >>> > >>> > > d[1:500,1] >>> > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [301] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0 0 0 0 0 0 0 0 0 0
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
Oops! Please ignore my last post. I mistakenly gave you different data I was testing with. This is the correct data: Here you go: > attach(as.data.frame(l_yx)) > range(x[y==0]) [1] 0.0 14.66518 > range(x[y==1]) [1] 0.0 13.49791 How do I know what is acceptable? Also, here are the screen shots of my data that I tried to send earlier (two screen shots, two pages): http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf Thank you, Ben On Thu, Dec 1, 2011 at 3:07 PM, Ben quant wrote: > Here you go: > > > attach(as.data.frame(l_yx)) > > range(x[y==1]) > [1] -22500.746. > > range(x[y==0]) > [1] -10076.5303653.0228 > > How do I know what is acceptable? > > Also, here are the screen shots of my data that I tried to send earlier > (two screen shots, two pages): > http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf > > Thank you, > > Ben > > > On Thu, Dec 1, 2011 at 2:24 PM, peter dalgaard wrote: > >> >> On Dec 1, 2011, at 21:32 , Ben quant wrote: >> >> > Thank you for the feedback, but my data looks fine to me. Please tell >> me if I'm not understanding. >> >> Hum, then maybe it really is a case of a transition region being short >> relative to the range of your data. Notice that the warning is just that: a >> warning. I do notice that the distribution of your x values is rather >> extreme -- you stated a range of 0--14 and a mean of 0.01. And after all, >> an odds ratio of 3000 per unit is only a tad over a doubling per 0.1 units. >> >> Have a look at range(x[y==0]) and range(x[y==1]). >> >> >> > >> > I followed your instructions and here is a sample of the first 500 >> values : (info on 'd' is below that) >> > >> > > d <- as.data.frame(l_yx) >> > > x = with(d, y[order(x)]) >> > > x[1:500] # I have 1's and 0's dispersed throughout >> > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 >> > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 >> 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 >> 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 >> > [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 >> 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 >> > >> > # I get the warning still >> > > l_df = as.data.frame(l_yx) >> > > l_logit = glm(y~x, data=l_df, family=binomial(link="logit")) >> > >> > Warning message: >> > glm.fit: fitted probabilities numerically 0 or 1 occurred >> > >> > # some info on 'd' above: >> > >> > > d[1:500,1] >> > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [301] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> > > d[1:500,2] >> > [1] 3.023160e-03 7.932130e-02 0.00e+00 1.779657e-02 1.608374e-01 >> 0.00e+00 5.577064e-02 7.753926e-03 4.018553e-03 4.760918e-02 >> 2.080511e-02 1.642404e-01 3.703720e-03 8.901981e-02 1.260415e-03 >> > [16] 2.202523e-02 3.750940e-02 4.441975e-04 9.351171e-03 8.374567e-03 >> 0.00e+00 8.440448e-02 5.081017e-01 2.538640e-05 1.806017e-02 >> 2.954641e-04 1.434859e-03 6.964976e-04 0.00e+00 1.202162e-02 >> > [31] 3.420300e-03 4.276100e-02 1.457324e-02 4.140121e-03 1.349180e-04 >> 1.525292e-03 4.817502e-02 9.515717e-03 2.232953e-02 1.227908e-01 >> 3.293581e-02 1.454352e-02 1.176011e-03 6.274138e-02 2.879205e-02 >> > [46] 6.900926e-03 1.414648e-04 3.446349e-02 8.807174e-03 3
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
Here you go: > attach(as.data.frame(l_yx)) > range(x[y==1]) [1] -22500.746. > range(x[y==0]) [1] -10076.5303653.0228 How do I know what is acceptable? Also, here are the screen shots of my data that I tried to send earlier (two screen shots, two pages): http://scientia.crescat.net/static/ben/warn%20num%200%20or%201.pdf Thank you, Ben On Thu, Dec 1, 2011 at 2:24 PM, peter dalgaard wrote: > > On Dec 1, 2011, at 21:32 , Ben quant wrote: > > > Thank you for the feedback, but my data looks fine to me. Please tell me > if I'm not understanding. > > Hum, then maybe it really is a case of a transition region being short > relative to the range of your data. Notice that the warning is just that: a > warning. I do notice that the distribution of your x values is rather > extreme -- you stated a range of 0--14 and a mean of 0.01. And after all, > an odds ratio of 3000 per unit is only a tad over a doubling per 0.1 units. > > Have a look at range(x[y==0]) and range(x[y==1]). > > > > > > I followed your instructions and here is a sample of the first 500 > values : (info on 'd' is below that) > > > > > d <- as.data.frame(l_yx) > > > x = with(d, y[order(x)]) > > > x[1:500] # I have 1's and 0's dispersed throughout > > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 > > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 > 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 > 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 > > [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 > 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > > > > # I get the warning still > > > l_df = as.data.frame(l_yx) > > > l_logit = glm(y~x, data=l_df, family=binomial(link="logit")) > > > > Warning message: > > glm.fit: fitted probabilities numerically 0 or 1 occurred > > > > # some info on 'd' above: > > > > > d[1:500,1] > > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [301] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > > d[1:500,2] > > [1] 3.023160e-03 7.932130e-02 0.00e+00 1.779657e-02 1.608374e-01 > 0.00e+00 5.577064e-02 7.753926e-03 4.018553e-03 4.760918e-02 > 2.080511e-02 1.642404e-01 3.703720e-03 8.901981e-02 1.260415e-03 > > [16] 2.202523e-02 3.750940e-02 4.441975e-04 9.351171e-03 8.374567e-03 > 0.00e+00 8.440448e-02 5.081017e-01 2.538640e-05 1.806017e-02 > 2.954641e-04 1.434859e-03 6.964976e-04 0.00e+00 1.202162e-02 > > [31] 3.420300e-03 4.276100e-02 1.457324e-02 4.140121e-03 1.349180e-04 > 1.525292e-03 4.817502e-02 9.515717e-03 2.232953e-02 1.227908e-01 > 3.293581e-02 1.454352e-02 1.176011e-03 6.274138e-02 2.879205e-02 > > [46] 6.900926e-03 1.414648e-04 3.446349e-02 8.807174e-03 3.549332e-02 > 2.828509e-03 2.935003e-02 7.162872e-03 5.650050e-03 1.221191e-02 > 0.00e+00 2.126334e-02 2.052020e-02 7.542409e-02 2.586269e-04 > > [61] 5.258664e-02 1.213126e-02 1.493275e-02 8.152657e-03 1.774966e-02 > 2.433198e-02 1.371257e-02 6.732339e-02 9.747677e-03 8.180594e-03 > 1.882473e-01 1.682596e-03 1.410170e-02 8.429676e-03 3.520589e-01 > > [76] 1.865160e-02 2.389941e-02 3.301558e-02 4.896605e-03 6.826244e-03 > 9.151337e-02 1.722270e-02 1.764039e-01 3.554063e-02 7.338532e-03 > 9.577793e-03 3.632366e-03 1.422400e-02 3.020497e-02 1.797096e-02 > > [91] 2.577899e-03 1.338365e-01 8.202223e-03 1.088335e-02
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
On Dec 1, 2011, at 21:32 , Ben quant wrote: > Thank you for the feedback, but my data looks fine to me. Please tell me if > I'm not understanding. Hum, then maybe it really is a case of a transition region being short relative to the range of your data. Notice that the warning is just that: a warning. I do notice that the distribution of your x values is rather extreme -- you stated a range of 0--14 and a mean of 0.01. And after all, an odds ratio of 3000 per unit is only a tad over a doubling per 0.1 units. Have a look at range(x[y==0]) and range(x[y==1]). > > I followed your instructions and here is a sample of the first 500 values : > (info on 'd' is below that) > > > d <- as.data.frame(l_yx) > > x = with(d, y[order(x)]) > > x[1:500] # I have 1's and 0's dispersed throughout > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 > 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 > 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 > [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 > > # I get the warning still > > l_df = as.data.frame(l_yx) > > l_logit = glm(y~x, data=l_df, family=binomial(link="logit")) > > Warning message: > glm.fit: fitted probabilities numerically 0 or 1 occurred > > # some info on 'd' above: > > > d[1:500,1] > [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [301] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > > d[1:500,2] > [1] 3.023160e-03 7.932130e-02 0.00e+00 1.779657e-02 1.608374e-01 > 0.00e+00 5.577064e-02 7.753926e-03 4.018553e-03 4.760918e-02 2.080511e-02 > 1.642404e-01 3.703720e-03 8.901981e-02 1.260415e-03 > [16] 2.202523e-02 3.750940e-02 4.441975e-04 9.351171e-03 8.374567e-03 > 0.00e+00 8.440448e-02 5.081017e-01 2.538640e-05 1.806017e-02 2.954641e-04 > 1.434859e-03 6.964976e-04 0.00e+00 1.202162e-02 > [31] 3.420300e-03 4.276100e-02 1.457324e-02 4.140121e-03 1.349180e-04 > 1.525292e-03 4.817502e-02 9.515717e-03 2.232953e-02 1.227908e-01 3.293581e-02 > 1.454352e-02 1.176011e-03 6.274138e-02 2.879205e-02 > [46] 6.900926e-03 1.414648e-04 3.446349e-02 8.807174e-03 3.549332e-02 > 2.828509e-03 2.935003e-02 7.162872e-03 5.650050e-03 1.221191e-02 0.00e+00 > 2.126334e-02 2.052020e-02 7.542409e-02 2.586269e-04 > [61] 5.258664e-02 1.213126e-02 1.493275e-02 8.152657e-03 1.774966e-02 > 2.433198e-02 1.371257e-02 6.732339e-02 9.747677e-03 8.180594e-03 1.882473e-01 > 1.682596e-03 1.410170e-02 8.429676e-03 3.520589e-01 > [76] 1.865160e-02 2.389941e-02 3.301558e-02 4.896605e-03 6.826244e-03 > 9.151337e-02 1.722270e-02 1.764039e-01 3.554063e-02 7.338532e-03 9.577793e-03 > 3.632366e-03 1.422400e-02 3.020497e-02 1.797096e-02 > [91] 2.577899e-03 1.338365e-01 8.202223e-03 1.088335e-02 2.137259e-02 > 6.829797e-03 3.104860e-04 8.927160e-03 5.626881e-02 1.197682e-02 6.140553e-02 > 2.729364e-02 1.377913e-02 3.018277e-03 1.188304e-02 > [106] 2.029268e-03 2.108815e-02 1.765503e-03 2.449253e-02 3.677046e-03 > 1.013463e-02 4.286642e-02 1.238931e-02 3.072090e-03 1.509613e-02 3.609885e-02 > 5.537268e-03 3.151614e-02 3.703148e-03 1.409401e-01 > [121] 1.473365e-02 9.160461e-03 1.035099e-01 3.005865e-02 1.332199e-02 > 6.936535e-03 2.433539e-0
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
Thank you for the feedback, but my data looks fine to me. Please tell me if I'm not understanding. I followed your instructions and here is a sample of the first 500 values : (info on 'd' is below that) > d <- as.data.frame(l_yx) > x = with(d, y[order(x)]) > x[1:500] # I have 1's and 0's dispersed throughout [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 [301] 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 # I get the warning still > l_df = as.data.frame(l_yx) > l_logit = glm(y~x, data=l_df, family=binomial(link="logit")) Warning message: glm.fit: fitted probabilities numerically 0 or 1 occurred # some info on 'd' above: > d[1:500,1] [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [101] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [201] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [301] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [401] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > d[1:500,2] [1] 3.023160e-03 7.932130e-02 0.00e+00 1.779657e-02 1.608374e-01 0.00e+00 5.577064e-02 7.753926e-03 4.018553e-03 4.760918e-02 2.080511e-02 1.642404e-01 3.703720e-03 8.901981e-02 1.260415e-03 [16] 2.202523e-02 3.750940e-02 4.441975e-04 9.351171e-03 8.374567e-03 0.00e+00 8.440448e-02 5.081017e-01 2.538640e-05 1.806017e-02 2.954641e-04 1.434859e-03 6.964976e-04 0.00e+00 1.202162e-02 [31] 3.420300e-03 4.276100e-02 1.457324e-02 4.140121e-03 1.349180e-04 1.525292e-03 4.817502e-02 9.515717e-03 2.232953e-02 1.227908e-01 3.293581e-02 1.454352e-02 1.176011e-03 6.274138e-02 2.879205e-02 [46] 6.900926e-03 1.414648e-04 3.446349e-02 8.807174e-03 3.549332e-02 2.828509e-03 2.935003e-02 7.162872e-03 5.650050e-03 1.221191e-02 0.00e+00 2.126334e-02 2.052020e-02 7.542409e-02 2.586269e-04 [61] 5.258664e-02 1.213126e-02 1.493275e-02 8.152657e-03 1.774966e-02 2.433198e-02 1.371257e-02 6.732339e-02 9.747677e-03 8.180594e-03 1.882473e-01 1.682596e-03 1.410170e-02 8.429676e-03 3.520589e-01 [76] 1.865160e-02 2.389941e-02 3.301558e-02 4.896605e-03 6.826244e-03 9.151337e-02 1.722270e-02 1.764039e-01 3.554063e-02 7.338532e-03 9.577793e-03 3.632366e-03 1.422400e-02 3.020497e-02 1.797096e-02 [91] 2.577899e-03 1.338365e-01 8.202223e-03 1.088335e-02 2.137259e-02 6.829797e-03 3.104860e-04 8.927160e-03 5.626881e-02 1.197682e-02 6.140553e-02 2.729364e-02 1.377913e-02 3.018277e-03 1.188304e-02 [106] 2.029268e-03 2.108815e-02 1.765503e-03 2.449253e-02 3.677046e-03 1.013463e-02 4.286642e-02 1.238931e-02 3.072090e-03 1.509613e-02 3.609885e-02 5.537268e-03 3.151614e-02 3.703148e-03 1.409401e-01 [121] 1.473365e-02 9.160461e-03 1.035099e-01 3.005865e-02 1.332199e-02 6.936535e-03 2.433539e-02 4.935373e-03 4.822740e-03 1.597643e-03 3.805619e-03 1.092683e-02 1.760635e-01 5.565614e-03 7.739213e-04 [136] 4.529198e-03 5.110359e-03 7.391258e-02 5.018207e-03 2.071417e-02 1.825787e-02 9.141405e-03 1.078386e-01 2.171470e-04 7.164583e-03 2.522077e-02 1.994428e-03 6.044653e-03 1.778078e-04 5.797706e-03 [151] 1.526778e-02 1.595897e-02 1.995627e-01 1.946735e-03 6.711582e-02 2.722350e-02 3.127499e-02 1.074904e-01 2.477414e-03 5.015375e-02 3.417810e-02 2.046643e-02 1.644832e-02 5.220166e-03 1.217752e-02 [166] 1.187352e-02 1.634016e-02 2.349568e-03 3.451811e-02 2.593540e-03 6.868595e-03 1.311236e-02 6.457757e-03 2.942391e-04 1.628352e-02 8.288831e-03 3.170856e-04
Re: [R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
On Dec 1, 2011, at 18:54 , Ben quant wrote: > Sorry if this is a duplicate: This is a re-post because the pdf's mentioned > below did not go through. Still not there. Sometimes it's because your mailer doesn't label them with the appropriate mime-type (e.g. as application/octet-stream, which is "arbitrary binary"). Anyways, see below [snip] > > With the above data I do: >>l_logit = glm(y~x, data=as.data.frame(l_yx), > family=binomial(link="logit")) > Warning message: > glm.fit: fitted probabilities numerically 0 or 1 occurred > > Why am I getting this warning when I have data points of varying values for > y=1 and y=0? In other words, I don't think I have the linear separation > issue discussed in one of the links I provided. I bet that you do... You can get the warning without that effect (one of my own examples is the probability of menarche in a data set that includes infants and old age pensioners), but not with a huge odds ratio as well. Take a look at d <- as.data.frame(l_yx) with(d, y[order(x)]) if it comes out as all zeros followed by all ones or vice versa, then you have the problem. > > PS - Then I do this and I get a odds ratio a crazy size: >>l_sm = summary(l_logit) # coef pval is $coefficients[8], log odds > $coefficients[2] >>l_exp_coef = exp(l_logit$coefficients)[2] # exponentiate the > coeffcients >>l_exp_coef > x > 3161.781 > > So for one unit increase in the predictor variable I get 3160.781% > (3161.781 - 1 = 3160.781) increase in odds? That can't be correct either. > How do I correct for this issue? (I tried multiplying the predictor > variables by a constant and the odds ratio goes down, but the warning above > still persists and shouldn't the odds ratio be predictor variable size > independent?) -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] logistic regression - glm.fit: fitted probabilities numerically 0 or 1 occurred
Sorry if this is a duplicate: This is a re-post because the pdf's mentioned below did not go through. Hello, I'm new'ish to R, and very new to glm. I've read a lot about my issue: Warning message: glm.fit: fitted probabilities numerically 0 or 1 occurred ...including: http://tolstoy.newcastle.edu.au/R/help/05/07/7759.html http://r.789695.n4.nabble.com/glm-fit-quot-fitted-probabilities-numerically-0-or-1-occurred-quot-td849242.html (note that I never found: "MASS4 pp.197-8" However, Ted's post was quite helpful.) This is a common question, sorry. Because it is a common issue I am posting everything I know about the issue and how I think I am not falling into the same trap at the others (but I must be due to some reason I am not yet aware of). >From the two links above I gather that my warning "glm.fit: fitted probabilities numerically 0 or 1 occurred" arises from a "perfect fit" situation (i.e. the issue where all the high value x's (predictor variables) are Y=1 (response=1) or the other way around). I don't feel my data has this issue. Please point out how it does! The list post instructions state that I can attach pdf's, so I attached plots of my data right before I do the call to glm. The attachments are plots of my data stored in variable l_yx (as can be seen in the axis names): My response (vertical axis) by row index (horizontal axis): plot(l_yx[,1],type='h') My predictor variable (vertical axis) by row index index (horizontal axis): plot(l_yx[,2],type='h') So here is more info on my data frame/data (in case you can't see my pdf attachments): > unique(l_yx[,1]) [1] 0 1 > mean(l_yx[,2]) [1] 0.01123699 > max(l_yx[,2]) [1] 14.66518 > min(l_yx[,2]) [1] 0 > attributes(l_yx) $dim [1] 690303 2 $dimnames $dimnames[[1]] NULL $dimnames[[2]] [1] "y" "x" With the above data I do: > l_logit = glm(y~x, data=as.data.frame(l_yx), family=binomial(link="logit")) Warning message: glm.fit: fitted probabilities numerically 0 or 1 occurred Why am I getting this warning when I have data points of varying values for y=1 and y=0? In other words, I don't think I have the linear separation issue discussed in one of the links I provided. PS - Then I do this and I get a odds ratio a crazy size: > l_sm = summary(l_logit) # coef pval is $coefficients[8], log odds $coefficients[2] > l_exp_coef = exp(l_logit$coefficients)[2] # exponentiate the coeffcients > l_exp_coef x 3161.781 So for one unit increase in the predictor variable I get 3160.781% (3161.781 - 1 = 3160.781) increase in odds? That can't be correct either. How do I correct for this issue? (I tried multiplying the predictor variables by a constant and the odds ratio goes down, but the warning above still persists and shouldn't the odds ratio be predictor variable size independent?) Thank you for your help! Ben [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
