Re: [R] difference between max in summary table and max function
I thought I'd chime in ... submitting the following: ?summary Provides the following documentation for the default for generalized argument (other than class=data.frame, factor, or matrix): ## Default S3 method: summary(object, ..., digits = max(3, getOption(digits)-3)) so passing along the object testrow w/o a corresponding argument for digits ... defaults to digits=4 (assuming your system has the same default option of digits = 7 that mine does). ... and since later in the documentation it indicates that digits is an: integer, used for number formatting with signif() so noting that all of the values you reported from summary(testrow) all have 4 significant digits (including the Max. of 131500) (excepting the min value of 1), summary() is doing what it is documented to do. ... sorry for being pedantic --- but doing so to point out how helpful the ? command can be sometimes. Hope this helps. __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com LinkedIn Profile: www.linkedin.com/pub/allen-bingham/3b/556/325 -Original Message- From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Martyn Byng Sent: Friday, February 13, 2015 3:15 AM To: Franckx Laurent; r-help@r-project.org Subject: Re: [R] difference between max in summary table and max function Its a formatting thing, try summary(testrow,digits=20) -Original Message- From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Franckx Laurent Sent: 13 February 2015 11:00 To: r-help@r-project.org Subject: [R] difference between max in summary table and max function Dear all I have found out that the max in the summary of an integer vector is not always equal to the actual maximum of that vector. For example: testrow - c(1:131509) summary(testrow) Min. 1st Qu. MedianMean 3rd Qu.Max. 1 32880 65760 65760 98630 131500 max(testrow) [1] 131509 This has occurred both in a Windows and in a Linux environment. Does this mean that the max value in the summary is only an approximation? Best regards Laurent Franckx, PhD Senior researcher sustainable mobility VITO NV | Boeretang 200 | 2400 Mol Tel. ++ 32 14 33 58 22| mob. +32 479 25 59 07 | Skype: laurent.franckx | laurent.fran...@vito.be | Twitter @LaurentFranckx VITO Disclaimer: http://www.vito.be/e-maildisclaimer __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. This e-mail has been scanned for all viruses by Star.\ _...{{dropped:8}} __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] SAS equivalent for R's signif function?
Daniel, Thanks for the link and the advice (vis-à-vis SAS-L ... after 30+ years of using SAS I should have remembered that one which I used extensively when learning SAS). FYI-I'm working on some R code that may eventually need to be ported to SAS --- and hence the reason for my question --- I don't want to use code in R that might be too difficult to replicate in SAS. I'll send you a separate email about the PROC FCMP implementation you mentioned. Thanks again-Allen -Original Message- From: Daniel Nordlund [mailto:djnordl...@frontier.com] Sent: Monday, February 9, 2015 11:05 PM To: Allen Bingham; r-help@r-project.org Subject: Re: [R] SAS equivalent for R's signif function? On 2/9/2015 7:02 PM, Allen Bingham wrote: Probably posting this to the wrong list ... but I'm in the process of learning R, after many years of using SAS --- so I thought I'd ask this question here: Is there with a function (or macro) in SAS that performs the same action as R's signif function, if so please provide? Tried to find via a Google search to no success. Doesn't seem to be in the R for SAS and SPSS Users by Robert A. Munchen (first edition is what I have), or in SAS and R by Ken Kleinman and Nicholas J. Horton (2nd edition) [although in the latter they do list the R signif function on page 61 ... but don't list a SAS equivalent. If you have a suggestion for a different list that I might ask this question (assuming I don't get the answer here), provide that as well. Thanks-Allen __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com LinkedIn Profile: www.linkedin.com/pub/allen-bingham/3b/556/325 __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Yes, this is the wrong list. Your question is about SAS (that is what SAS-L or SAScommunity is about) and this list is about R, and you already know what function to use in R. That said, I searched for SAS round to fixed number of significant digits and found this link http://support.sas.com/kb/24/728.html You could turn this into a function style macro (but it will be ugly). If you have a recent enough version of SAS you could use PROC FCMP to turn this into a function. If you are interested contact me offline and I will send you a PROC FCMP implementation. Dan -- Daniel Nordlund Bothell, WA USA __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Error: OutOfMemoryError (Java): GC overhead limit exceeded
Rebecca, An additional issue might be plots, pivot tables, etc. that you might have in separate sheets on the input Excel workbook (if you have those, just make a separate workbook with only the data you need for R in it). BTW---I've recently been using the following option for similar use of XLConnect: options(java.parameters = -Xmx4g ) to set it at 4 Gbytes ... but I have lots of memory on my machine (16 Gbytes). Another issue is that the above setting is for one instance/session of R --- you can't re-issue it with a larger number later in the same session --- you have to quit R first then re-issue with a larger number. Good luck-Allen __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com LinkedIn Profile: www.linkedin.com/pub/allen-bingham/3b/556/325 -Original Message- From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of jim holtman Sent: Monday, February 2, 2015 10:23 AM To: Yuan, Rebecca Cc: R help (r-help@r-project.org) Subject: Re: [R] Error: OutOfMemoryError (Java): GC overhead limit exceeded How big are the worksheets that you are reading in? Do you have multiple ones open at the same time? Have to tried to use 'xlcFreeMemory' to see if this helps? How much RAM to you have on your system? Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Mon, Feb 2, 2015 at 1:11 PM, Yuan, Rebecca rebecca.y...@bankofamerica.com wrote: Hello Jim, I already use “.xls” for the loading, but still have the memory issue…. Thanks, Rebecca *From:* jim holtman [mailto:jholt...@gmail.com] *Sent:* Monday, February 02, 2015 10:21 AM *To:* Jeff Newmiller *Cc:* Yuan, Rebecca; R help (r-help@r-project.org) *Subject:* Re: [R] Error: OutOfMemoryError (Java): GC overhead limit exceeded On the off-chance, are you using XLConnect or xlsx packages that are using Java to access the spreadsheets? If it is XLConnect, are you access the '.xlsx' style workbooks? If so, can you try using '.xls' workbooks? This is a problem that I have had; the '.xlsx' workbooks take a lot more resources (CPU and memory) to process. A little more information like what your sessionInfo is would help a lot in responding to your problem. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Mon, Feb 2, 2015 at 9:27 AM, Jeff Newmiller jdnew...@dcn.davis.ca.us wrote: The memory issue is in Java, not R. You can either be more parsimonious in your use of Java memory (we have no idea what you are doing with it here, so how you do that is up to you), or you can allocate more memory to Java (you may be able to guess how to do that, or read the Java documentation on the X parameter). --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k -- - Sent from my phone. Please excuse my brevity. On February 2, 2015 5:43:50 AM PST, Yuan, Rebecca rebecca.y...@bankofamerica.com wrote: Hello all, When I met this following error message: Error: OutOfMemoryError (Java): GC overhead limit exceeded I usually use the following options to overcome the memory limit: options(java.parameters = -Xmx1024m) # to reduce the error message Error: OutOfMemoryError (Java): GC overhead limit exceeded However, this seems not working any more, is there any other way to help avoiding the memory error issue in R? Thanks very much! Cheers, Rebecca 20 - - This message, and any attachments, is for the intended r...{{dropped:5}} __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- This message, and any attachments, is for the \ in...{{dropped:10}} __ R
[R] SAS equivalent for R's signif function?
Probably posting this to the wrong list ... but I'm in the process of learning R, after many years of using SAS --- so I thought I'd ask this question here: Is there with a function (or macro) in SAS that performs the same action as R's signif function, if so please provide? Tried to find via a Google search to no success. Doesn't seem to be in the R for SAS and SPSS Users by Robert A. Munchen (first edition is what I have), or in SAS and R by Ken Kleinman and Nicholas J. Horton (2nd edition) [although in the latter they do list the R signif function on page 61 ... but don't list a SAS equivalent. If you have a suggestion for a different list that I might ask this question (assuming I don't get the answer here), provide that as well. Thanks-Allen __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com LinkedIn Profile: www.linkedin.com/pub/allen-bingham/3b/556/325 __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] censored quantile regression
Ashley, Looking at the code for the crq function it appears that with method=Portnoy the tau value is not passed to the fitting function (crq.fit.por) and that function appears (I didn't delve into the code to make sure of this ...) to calculate the grid of tau values given the nature of the data, formula, etc. --- --- more telling though is the following snippet I copied from the documentation of the crq function, specifically it states (in part) [I've split apart the pertinent section first]: ... Both the Portnoy and Peng-Huang estimators may be unable to compute estimates of the conditional quantile parameters in the upper tail of distribution. ... Like the Kaplan-Meier estimator, when censoring is heavy in the upper tail the estimated distribution is defective and quantiles are only estimable on a sub-interval of (0,1). The Peng and Huang estimator can be viewed as a generalization of the Nelson Aalen estimator of the cumulative hazard function, and can be formulated as a variant of the conventional quantile regression dual problem. See Koenker (2008) for further details. This paper is available from the package with vignette(crq). ... ... so I'm guessing that using one of the other fitting methods may be get you what you need --- HOWEVER I'm only looking at the underlying code and documentation for same --- I'm completely ignorant of the statistical methods you are trying to implement --- and whether or not one of the other fitting methods is appropriate to your analyses. Regardless it seems to me (IMHO) that the documentation for the crq function should indicate that the tau or alternatively the taus supplied to the crq function are only used with a subset of the method values. Hope this helps-Allen __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com This message and any attachments may contain confidential or privileged information and are only for the use of the intended recipient of this message. If you are not the intended recipient, please notify the sender by return email, and delete or destroy this and all copies of this message and all attachments. Any unauthorized disclosure, use, distribution, or reproduction of this message or any attachments without permission from the sender is prohibited and may be unlawful. -Original Message- From: Ashley Isaac Naimi, Mr [mailto:ashley.na...@mcgill.ca] Sent: Tuesday, January 20, 2015 7:52 AM To: r-help@r-project.org Subject: [R] censored quantile regression Hi there, I've generated the following data: library(quantreg) library(survival) set.seed(789) N - 2000 u - runif(N) x1 - rbinom(N,1,.5) x2 - rbinom(N,1,.5) x1x2-x1*x2 lambda - 1 + 1.5*x1 + 1.5*x2 + .5*x1x2 k - 2 y - lambda*((-log(1-u))^(1/k));max(y) c - runif(N,max=15) event = as.numeric(y=c) mean(event);table(event) cens - 1-event table(cens);mean(cens) time -as.matrix(ifelse(event==1,y,c)) St-Surv(time,event,type=right) To which I've fit the following censored quantile regression model: q2 - crq(St~x1 + x2 + x1x2,tau=.9,method=Portnoy) summary(q2) As one can see, I'm interested in the 0.9th quantile. But summary(q2) returns only the 20th to 80th percentiles (by 20). How can I get only the 0.9th quantile (aka the 90th percentile)?? All I actually need is the parameter estimate for the interaction (x1x2) for the 90th percentile. I know how to extract it if I were able to obtain estimates at tau=0.9, but no luck. Even though I request the 90th percentile in crq (i.e., tau=0.9), the summary function keeps returning the same set of (unwanted) percentiles (20th, 40th, 60th, 80th). I’m using R version 3.1.2 (2014-10-31) -- Pumpkin Helmet”, Platform: x86_64-apple-darwin13.4.0 (64-bit). Cheers, Ashley I Naimi [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] pch size in a legend
Could you provide an example of trying to use pt.cex that does not do the job? Using pt.cex works fine for me when I've used it (R version 3.1.2 both 32 bit and 64 bit). Here's some dummy code that demonstrates that the symbol size changes w/o changing the text size: plot(x=c(0,6),y=c(0,6),type=n) legend(x=bottomright,title=Legend 1, symbol 2 with pt.cex=1.25,legend=c(Item 1,Item 2),pch=c(1,1),pt.cex=c(1,1.25),col=c('black','blue')) legend(x=topleft,title=Legend 2, symbol 2 with pt.cex=1.75,legend=c(Item 1,Item 2),pch=c(1,1),pt.cex=c(1,1.75),col=c('black','blue')) see if that works for you? __ Allen Bingham Bingham Statistical Consulting aebingh...@gmail.com -Original Message- From: Ahmed Attia [mailto:ahmedati...@gmail.com] Sent: Friday, January 30, 2015 1:50 PM To: r-help Subject: [R] pch size in a legend Hi R users, I would like to adjust the pch size in a legend without changing the text size, pt.cex does not do the job. R 2.15.2 32 bit. legend(0,2100, legend=c(2009,2010,2012,2013,2014), col = 1,cex=1,lty=NA,pch=c(1,2,6,7,8),lwd=2,bty=n) Thanks Ahmed Attia, Ph.D. Agronomist Soil Scientist Post-Doc Research Associate Texas AM AgriLife Research-Vernon ahmed.at...@ag.tamu.edu Cell phone: 001-979-248-5215 __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Sum function and missing values --- need to mimic SAS sum function
Sven and John, Thanks for your suggested code ... hits the mark! The code by John is what I need to be able to use in an apply function, but I really like the simplicity of Sven's suggestion. Also thanks to all who replied --- really helped broaden my knowledge of R. Allen -Original Message- From: Sven E. Templer [mailto:sven.temp...@gmail.com] Sent: Monday, January 26, 2015 6:56 AM To: Martin Maechler Cc: Jim Lemon; r-help mailing list; Allen Bingham Subject: Re: [R] Sum function and missing values --- need to mimic SAS sum function you can also define 'na.rm' in sum() by 'NA state' of x (where x is your vector holding the data): sum(x, na.rm=!all(is.na(x))) On 26 January 2015 at 13:45, Martin Maechler maech...@lynne.stat.math.ethz.ch wrote: Jim Lemon drjimle...@gmail.com on Mon, 26 Jan 2015 11:21:03 +1100 writes: Hi Allen, How about this: sum_w_NA-function(x) ifelse(all(is.na(x)),NA,sum(x,na.rm=TRUE)) Excuse, Jim, but that's yet another horrible misuse of ifelse() John Fox's reply *did* contain the proper solution if (all(is.na(x))) NA else sum(x, na.rm=TRUE) The ifelse() function should never be used in such cases. Read more after googling Do NOT use ifelse() -- include the quotes in your search -- or directly at http://stat.ethz.ch/pipermail/r-help/2014-December/424367.html Yes, this has been on R-help a month ago.. Martin On Mon, Jan 26, 2015 at 10:21 AM, Allen Bingham aebingh...@gmail.com wrote: I understand that in order to get the sum function to ignore missing values I need to supply the argument na.rm=TRUE. However, when summing numeric values in which ALL components are NA ... the result is 0.0 ... instead of (what I would get from SAS) of NA (or in the case of SAS .). Accordingly, I've had to go to 'extreme' measures to get the sum function to result in NA if all arguments are missing (otherwise give me a sum of all non-NA elements). So for example here's a snippet of code that ALMOST does what I want: SumValue-apply(subset(InputDataFrame,!is.na(Variable.1)|!is.na(Variable.2), select=c(Variable.1,Variable.2)),1,sum,na.rm=TRUE) In reality this does NOT give me records with NA for SumValue ... but it doesn't give me values for any records in which both Variable.1 and Variable.2 are NA --- which is good enough for my purposes. I'm guessing with a little more work I could come up with a way to adapt the code above so that I could get it to work like SAS's sum function ... ... but before I go that extra mile I thought I'd ask others if they know of functions in either base R ... or in a package that will better mimic the SAS sum function. Any suggestions? Thanks. __ Allen Bingham aebingh...@gmail.com __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Sum function and missing values --- need to mimic SAS sum function
Don, The default for the sum function is to NOT remove NA before summing (i.e., option na.rm=FALSE), here's the results with na.rm=TRUE sum(NA,na.rm=TRUE) [1] 0 sum(c(NA,NA),na.rm=TRUE) [1] 0 sum(rep(NA,10),na.rm=TRUE) [1] 0 sum(as.numeric(letters[1:4]),na.rm=TRUE) [1] 0 Warning message: NAs introduced by coercion Hope that explains it a bit better. Others have replied with suggested solutions to my 'problem', and the one by John Fox is what I need (an actual function that I can use in an apply statement), although the suggested code by Sven Templer is appealing in its simplicity. Allen -Original Message- From: MacQueen, Don [mailto:macque...@llnl.gov] Sent: Monday, January 26, 2015 1:03 PM To: Allen Bingham; r-help@r-project.org Subject: Re: [R] Sum function and missing values --- need to mimic SAS sum function I'm a little puzzled by the assertion that the result is 0.0 when all the elements are NA: sum(NA) [1] NA sum(c(NA,NA)) [1] NA sum(rep(NA, 10)) [1] NA sum(as.numeric(letters[1:4])) [1] NA Warning message: NAs introduced by coercion Considering that the example snippet of code has several other aspects besides using sum(), among them subsetting rows of a data frame when there are apparently NAs in some its variables ... I wonder if the reason for the failure of that snippet has been misunderstood? -- Don MacQueen Lawrence Livermore National Laboratory 7000 East Ave., L-627 Livermore, CA 94550 925-423-1062 On 1/25/15, 3:21 PM, Allen Bingham aebingh...@gmail.com wrote: I understand that in order to get the sum function to ignore missing values I need to supply the argument na.rm=TRUE. However, when summing numeric values in which ALL components are NA ... the result is 0.0 ... instead of (what I would get from SAS) of NA (or in the case of SAS .). Accordingly, I've had to go to 'extreme' measures to get the sum function to result in NA if all arguments are missing (otherwise give me a sum of all non-NA elements). So for example here's a snippet of code that ALMOST does what I want: SumValue-apply(subset(InputDataFrame,!is.na(Variable.1)|!is.na(Variabl e.2 ), select=c(Variable.1,Variable.2)),1,sum,na.rm=TRUE) In reality this does NOT give me records with NA for SumValue ... but it doesn't give me values for any records in which both Variable.1 and Variable.2 are NA --- which is good enough for my purposes. I'm guessing with a little more work I could come up with a way to adapt the code above so that I could get it to work like SAS's sum function ... ... but before I go that extra mile I thought I'd ask others if they know of functions in either base R ... or in a package that will better mimic the SAS sum function. Any suggestions? Thanks. __ Allen Bingham aebingh...@gmail.com __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Sum function and missing values --- need to mimic SAS sum function
I understand that in order to get the sum function to ignore missing values I need to supply the argument na.rm=TRUE. However, when summing numeric values in which ALL components are NA ... the result is 0.0 ... instead of (what I would get from SAS) of NA (or in the case of SAS .). Accordingly, I've had to go to 'extreme' measures to get the sum function to result in NA if all arguments are missing (otherwise give me a sum of all non-NA elements). So for example here's a snippet of code that ALMOST does what I want: SumValue-apply(subset(InputDataFrame,!is.na(Variable.1)|!is.na(Variable.2), select=c(Variable.1,Variable.2)),1,sum,na.rm=TRUE) In reality this does NOT give me records with NA for SumValue ... but it doesn't give me values for any records in which both Variable.1 and Variable.2 are NA --- which is good enough for my purposes. I'm guessing with a little more work I could come up with a way to adapt the code above so that I could get it to work like SAS's sum function ... ... but before I go that extra mile I thought I'd ask others if they know of functions in either base R ... or in a package that will better mimic the SAS sum function. Any suggestions? Thanks. __ Allen Bingham aebingh...@gmail.com __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.