Re: [R] about data manipulation
Hi lily, If you want to use aggregate, supply the name of the function: aggregate(flow~year, data=df, "sum") You can also use "by" like this by(df$flow,df$year,FUN=sum) I assume that you don't have to worry about missing months in a year. Jim : On Thu, Dec 1, 2016 at 3:06 PM, lily liwrote: > Hi R users, > > I'm trying to manipulate dataset, but met some difficulties. > > df > year month flow > 2006 33.5 > 2006 43.8 > 2006 521 > 2006 632 > 2007 34.1 > 2007 44.4 > ... > > I want to calculate total flow for each year, and use the code below: > aggregate(flow~year, data=df, sum) > But it gave the error message: > Error in get(as.character(FUN), mode = "function", envir = envir) : > object 'FUN' of mode 'function' was not found > > What is the problem and how to solve it? Thanks for your help. > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] about data manipulation
> On 1 Dec 2016, at 05:06, lily liwrote: > > Hi R users, > > I'm trying to manipulate dataset, but met some difficulties. > > df > year month flow > 2006 33.5 > 2006 43.8 > 2006 521 > 2006 632 > 2007 34.1 > 2007 44.4 > ... > > I want to calculate total flow for each year, and use the code below: > aggregate(flow~year, data=df, sum) > But it gave the error message: > Error in get(as.character(FUN), mode = "function", envir = envir) : > object 'FUN' of mode 'function' was not found > > What is the problem and how to solve it? Thanks for your help. > Not enough information. If I try this df <- read.table(text="year month flow 2006 33.5 2006 43.8 2006 521 2006 632 2007 34.1 2007 44.4 ", header=TRUE) df aggregate(flow~year, data=df, sum) I get a correct answer. So you are likely doing something weird and not showing us all. > [[alternative HTML version deleted]] > Plan text mail. Has been asked many, many times before. Berend Hasselman > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Command for simple effects following 2 way anova with interaction
## Use the split argument to summary.aov ## This tests the levels of tension within each level of wool using a common ## Residuals sum of squares. sapply(warpbreaks, levels) model2 <- aov(breaks ~ wool/tension, data = warpbreaks) colnames(model.matrix(model2)) ## [1] "(Intercept)""woolB" "woolA:tensionM" "woolB:tensionM" ## [5] "woolA:tensionH" "woolB:tensionH" sapply(warpbreaks[2:3], levels) summary(model2, split=list("wool:tension"=list("woolA/tension"=c(1,3), "woolB/tension"=c(2,4 ## The less good choice has the same numerator sums of squares but ## different Residuals sums of squares, with fewer df for each, ## and comes to different conclusions. model3A <- aov(breaks ~ tension, data = warpbreaks[warpbreaks$wool=="A",]) summary(model3A) model3B <- aov(breaks ~ tension, data = warpbreaks[warpbreaks$wool=="B",]) summary(model3B) ## The boxplots show that model2 better describes the data. ## There is a significant difference for woolA and not for woolB. library(lattice) bwplot(breaks ~ tension | wool, data=warpbreaks) On Wed, Nov 30, 2016 at 10:55 PM, Ashim Kapoorwrote: > Dear All, > > Suppose I do :- > > head(warpbreaks) > model1<- aov(breaks ~ wool*tension,data = warpbreaks) > summary(model1) > > There is significant interaction. So I need to test for simple effects of > wool at each level of tension and vice versa. I can do a subset and then > do a one way anova for each level of tension and wool. My query is that is > there any command programmed which can do this automatically? > > Best Regards, > Ashim > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] about data manipulation
Hi R users, I'm trying to manipulate dataset, but met some difficulties. df year month flow 2006 33.5 2006 43.8 2006 521 2006 632 2007 34.1 2007 44.4 ... I want to calculate total flow for each year, and use the code below: aggregate(flow~year, data=df, sum) But it gave the error message: Error in get(as.character(FUN), mode = "function", envir = envir) : object 'FUN' of mode 'function' was not found What is the problem and how to solve it? Thanks for your help. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Command for simple effects following 2 way anova with interaction
Dear All, Suppose I do :- head(warpbreaks) model1<- aov(breaks ~ wool*tension,data = warpbreaks) summary(model1) There is significant interaction. So I need to test for simple effects of wool at each level of tension and vice versa. I can do a subset and then do a one way anova for each level of tension and wool. My query is that is there any command programmed which can do this automatically? Best Regards, Ashim [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Complex Survey MSE for prediction with ML
Hello, I have been toying with the survey package's withReplicates function, which lets users easily extend the survey package to support any weighted statistic. There are a number of ML algorithms in various packages that accept weights, and it is fairly easy to use them with withReplicates. Below is a naïve example: library(survey) library(rpart) library(gbm) data(api) # create survey object dstrat<-svydesign(id=~1,strata=~stype, weights=~pw, data=apistrat, fpc=~fpc) rstrat<-as.svrepdesign(dstrat) # try rpart predr <- as.data.frame(withReplicates(rstrat, function(w, data) { predict(rpart(api00~ell+meals+mobility,data=data,weights=w)) })) # try gbm predg <- as.data.frame(withReplicates(rstrat, function(w, data) { predict(gbm(api00~ell+meals+mobility,data=data,weights=w, n.trees=100)) })) # try regular svyglm preds <- as.data.frame(predict(svyglm(api00~ell+meals+mobility,rstrat))) head(data.frame(predr,predg,preds)) With rpart, the standard errors are absurdly large, and clearly incorrect. With gbm, the results seem reasonable. I see in this extremely old post that you can't use quantile regression with withReplicates for some survey designs and expect to get reasonable results: https://stat.ethz.ch/pipermail/r-help/2008-August/171620.html Quantiles and survey stats are messy business so that issue may be unique to quantile regressions, but based on that post it would seem that the function, and survey design need to have certain properties for withReplicates to generate valid SEs. This is not documented with withReplicates though. So my question is, what properties does an ML algorithm/survey design need for withReplicates to generate valid SEs? Kind Regards, Carl Ganz __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] transpose rows and columns for large data
On Tue, 29 Nov 2016 17:22:47 + (UTC) Elham - via R-helpwrote: > Is there another way (I prefer a way with Excel)? Search on "friends don't let friends use excel for statistics." Spreadsheets are an inherently perilous way to do statistics and Excel specifically is notoriously poor. In fact the reason I originally began using dedicated statistical packages (STATA first and now R) is that spreadsheets (Excel in my case) can throw subtle errors that can create problems. immediately, or even worse - later. I had Excel return a negative variance. Since variance is a squared value, unless you are dealing with some very exotic numbers including imaginary values, a negative variance is an absurd result. Further investigation revealed that other stat routines provided with Excel at the time were also throwing errors that could look reasonable and thus be missed. It was also simply using erroneously constructed methods and providing outright wrong results to things like Chi-square calculations. -- John __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] stringi behaves differently in 2 similar situations
A dot is treated differently if it has a number on no, one, or both sides. > stri_extract_all_words("me.com", simplify = TRUE) [,1] [1,] "me.com" > stri_extract_all_words("me1.com", simplify = TRUE) [,1] [,2] [1,] "me1" "com" > stri_extract_all_words("me1.2com", simplify = TRUE) [,1] [1,] "me1.2com" ?stri_extract_all_words sent me to ?"stringi-search-boundaries" which suggests that you should spend some time with the user guide: _Boundary Analysis_ - ICU User Guide, http://userguide.icu-project.org/boundaryanalysis> Depending on your objective, you might be better off with strsplit() separating on whitespace. Sarah On Wed, Nov 30, 2016 at 3:51 PM, Dimitri Liakhovitskiwrote: > Hello! > > library(stringi) > > stri_extract_all_words("me.com", simplify = TRUE) # returns with a dot > stri_extract_all_words("watch32.com", simplify = TRUE) # removes the dot > > Why is the dot removed only in the second case? > How is it possible to ask it NOT to remove the dot in the second case? > > Thanks a lot! > -- Sarah Goslee http://www.functionaldiversity.org __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] stringi behaves differently in 2 similar situations
Hello! library(stringi) stri_extract_all_words("me.com", simplify = TRUE) # returns with a dot stri_extract_all_words("watch32.com", simplify = TRUE) # removes the dot Why is the dot removed only in the second case? How is it possible to ask it NOT to remove the dot in the second case? Thanks a lot! -- Dimitri Liakhovitski __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Question regarding Naive Bayes
Hello, I am working with the naïve bayes function inlibrary(e1071). The function calls are: transactions.train.nb = naiveBayes(as.factor(DealerID) ~ as.factor(Manufacturer) + as.factor(RangeDesc) +as.factor(BodyType) +as.factor(FuelType) +as.factor(PaintColour) +as.factor(TransmissionType) +as.factor(Mileage) +as.factor(Registration), data=transactions.train, na.action=na.omit) where transactions.train is a dataframe with dimension 2032rows by 14 columns. and transactions.test.nb = predict(transactions.train.nb,transactions.test[,-1], type='raw') An example of the result are View(transactions.test.nb) Reduced results shown: 188 225 229 270 273 1 0.000984 0.000492 0.000492 0.000492 0.001476 2 0.000984 0.000492 0.000492 0.000492 0.001476 3 0.000984 0.000492 0.000492 0.000492 0.001476 4 0.000984 0.000492 0.000492 0.000492 0.001476 5 0.000984 0.000492 0.000492 0.000492 0.001476 I was struggling to understand why the returnedprobabilities are the same for each column as I was hoping for them to bedifferent. Dealer ID should have a different probability to row 1 than row 2.Each row does sum to 1. Transactions.train represents 67% of the full set of data. I’ve tried introducing laplace smoothing, and experimentedwith increasing and decreasing the number of parameters used to generate thetraining naivebayes object But as of yet I can’t figure it out. Could anybody help? Kind regards, Phil, [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Meta-Analysis using RVE (robumeta)
Hello, I'm runnig a meta-analysis, correcting for clustering of standard errors with the robumetacommand. R gives the following warning: If df < 4, do not trust the results In some cases my df is in fact <4. What does this tell me then? Does it simply mean that the number of included studies is likely too small (often aroung 5-6, but with more than 10 single effect sizes)? Is there anything else I could do for these cases? Thank you [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Improve performance of RConnection and Rserve
Hi all, If we execute an R script directly in R prompt in Linux environment, it takes around 30 mins to complete. The same R script if we execute from java using RConnection, eval and Rserve for each line it takes thrice the time. Basically the performance is very bad. And in most of the occasion it never ends up in result. Can anyone please tell how to improve the performance using Rconnection. Thanks in advance Ranjana [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Breaking down a list into a table
The tidyr::separate function modifies data frames. You did not give it a data frame. Re-read your preferred R introductory material on what a data frame is (R comes with "Introduction to R"). You also did not give a reproducible example... e.g. does not execute as-is in a clean R work space. library(tidyr) input <- data.frame( kop=rep( "w;d;e;f", 5 ) ) output <- separate( input, kop, into=c( "C1", "C2", "C3", "C4" ), sep=";" ) You really should read the documentation for any contributed package you use. (This presumes that you keep track of the fact that you are using them... which is a good idea as there are thousands of them). I recommend vignette("tidy-data") and ?separate in this case. -- Sent from my phone. Please excuse my brevity. On November 30, 2016 7:18:19 AM PST, Ferri Leberlwrote: > >Thank to Ulrik for the hint. >However, I don't comprehend the function until now: > >For example I made up an array "input": > >input > kop >[1,] "w;d;e;f" >[2,] "w;d;e;f" >[3,] "w;d;e;f" >[4,] "w;d;e;f" >[5,] "w;d;e;f" > >and tried to break it into four cols with commmand: > >output<-separate(into,kop,into=c("a","b","c","d"),sep=";") > >R returned: > >Fehler in UseMethod("separate_") : >nicht anwendbare Methode für 'separate_' auf Objekt der Klasse >"c('matrix', 'character')" angewendet > >Could you please explain me my mistake? > >Thank you in advance! > >Yours, Ferri > > > > > > > >Gesendet: Dienstag, 22. November 2016 um 14:57 Uhr >Von: "Ulrik Stervbo" >An: "Ferri Leberl" , "r-helpr-project.org" > >Betreff: Re: [R] Breaking down a list into a table > >Hi Ferri, > It sounds like the function 'separate' from the tidyr package is what >you look for, > >HTH >Ulrik > >On Tue, 22 Nov 2016 at 14:49 Ferri Leberl > wrote: > >Dear All, >I asked for support to deal with a hirarchy within a character >separated list. >I solved the problem crudely but effectively by > >- Choosing for a TSV as input, where in columns that may contain >several (or as well no) items the items are separated via semicolon >- adding semicolons to the first row to grant that the first row has >the maximum number of semicolons of this column >- grasping the column(x<-myarray[,y], where y is some integer value) >and saving it as a TSV (with only one column) >- importing it again, defining it semicolumn-separated, with fill >option > >To all those who feel pain reading this: Is there a shortcut? >Thank you in advance. >Yours, Ferri > >__ >R-help@r-project.org[mailto:R-help@r-project.org] mailing list -- To >UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help[https://stat.ethz.ch/mailman/listinfo/r-help] >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html[http://www.R-project.org/posting-guide.html] >and provide commented, minimal, self-contained, reproducible code. > >__ >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Breaking down a list into a table
Well, if i understand correctly, there's really no need to use special packages for such simple tasks. The strsplit() function in base R does nicely (see ?strplit): Here is a reproducible example, --which you did not provide; please do so in future: > testdat <- c("abc;def;hij","qq;sdff;kuqw","oprrt;lbas;rw") > testdat [1] "abc;def;hij" "qq;sdff;kuqw" "oprrt;lbas;rw" ## note that this is a vector, not an array/data frame with 1 column. You can subscript it in the call below -- e.g. testdat[,1] -- if it is an array. > out <- strsplit(testdat,split= ";") > out [[1]] [1] "abc" "def" "hij" [[2]] [1] "qq" "sdff" "kuqw" [[3]] [1] "oprrt" "lbas" "rw" ## Note that the output of strsplit() is a list because it is designed to handle the more general case of unequal numbers of pieces in each row. To reform it to a matrix (i.e. 2d array) use rbind in a do.call (?do.call) construct: > do.call(rbind,out) [,1][,2] [,3] [1,] "abc" "def" "hij" [2,] "qq""sdff" "kuqw" [3,] "oprrt" "lbas" "rw" All of this presupposes familiarity with basic R data structures and constructs. If you do not have such familiarity yet, then I think you would do well to go through some R tutorials to gain it (if you wish to continue to use R). Packages are nice, with many helpful features, but you still need to know the basics of the language -- and R *is* a language. As far as your error message is concerned, separate() is apparently an S3 generic function (more reading!) but has no method for matrices. If you had done the call as: output<-separate(into,input[,1] ,into=c("a","b","c","d"),sep=";") ## giving the data as a vector rather than an array of 1 column it probably would have worked (but you need to check, as I don't use the tidyr package). Cheers, -- Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Wed, Nov 30, 2016 at 7:18 AM, Ferri Leberlwrote: > > Thank to Ulrik for the hint. > However, I don't comprehend the function until now: > > For example I made up an array "input": > > input > kop > [1,] "w;d;e;f" > [2,] "w;d;e;f" > [3,] "w;d;e;f" > [4,] "w;d;e;f" > [5,] "w;d;e;f" > > and tried to break it into four cols with commmand: > > output<-separate(into,kop,into=c("a","b","c","d"),sep=";") > > R returned: > > Fehler in UseMethod("separate_") : > nicht anwendbare Methode für 'separate_' auf Objekt der Klasse "c('matrix', > 'character')" angewendet > > Could you please explain me my mistake? > > Thank you in advance! > > Yours, Ferri > > > > > > > > Gesendet: Dienstag, 22. November 2016 um 14:57 Uhr > Von: "Ulrik Stervbo" > An: "Ferri Leberl" , "r-helpr-project.org" > > Betreff: Re: [R] Breaking down a list into a table > > Hi Ferri, > It sounds like the function 'separate' from the tidyr package is what you > look for, > > HTH > Ulrik > > On Tue, 22 Nov 2016 at 14:49 Ferri Leberl > wrote: > > Dear All, > I asked for support to deal with a hirarchy within a character separated list. > I solved the problem crudely but effectively by > > - Choosing for a TSV as input, where in columns that may contain several (or > as well no) items the items are separated via semicolon > - adding semicolons to the first row to grant that the first row has the > maximum number of semicolons of this column > - grasping the column(x<-myarray[,y], where y is some integer value) and > saving it as a TSV (with only one column) > - importing it again, defining it semicolumn-separated, with fill option > > To all those who feel pain reading this: Is there a shortcut? > Thank you in advance. > Yours, Ferri > > __ > R-help@r-project.org[mailto:R-help@r-project.org] mailing list -- To > UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help[https://stat.ethz.ch/mailman/listinfo/r-help] > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html[http://www.R-project.org/posting-guide.html] > and provide commented, minimal, self-contained, reproducible code. > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Breaking down a list into a table
Thank to Ulrik for the hint. However, I don't comprehend the function until now: For example I made up an array "input": input kop [1,] "w;d;e;f" [2,] "w;d;e;f" [3,] "w;d;e;f" [4,] "w;d;e;f" [5,] "w;d;e;f" and tried to break it into four cols with commmand: output<-separate(into,kop,into=c("a","b","c","d"),sep=";") R returned: Fehler in UseMethod("separate_") : nicht anwendbare Methode für 'separate_' auf Objekt der Klasse "c('matrix', 'character')" angewendet Could you please explain me my mistake? Thank you in advance! Yours, Ferri Gesendet: Dienstag, 22. November 2016 um 14:57 Uhr Von: "Ulrik Stervbo"An: "Ferri Leberl" , "r-helpr-project.org" Betreff: Re: [R] Breaking down a list into a table Hi Ferri, It sounds like the function 'separate' from the tidyr package is what you look for, HTH Ulrik On Tue, 22 Nov 2016 at 14:49 Ferri Leberl wrote: Dear All, I asked for support to deal with a hirarchy within a character separated list. I solved the problem crudely but effectively by - Choosing for a TSV as input, where in columns that may contain several (or as well no) items the items are separated via semicolon - adding semicolons to the first row to grant that the first row has the maximum number of semicolons of this column - grasping the column(x<-myarray[,y], where y is some integer value) and saving it as a TSV (with only one column) - importing it again, defining it semicolumn-separated, with fill option To all those who feel pain reading this: Is there a shortcut? Thank you in advance. Yours, Ferri __ R-help@r-project.org[mailto:R-help@r-project.org] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help[https://stat.ethz.ch/mailman/listinfo/r-help] PLEASE do read the posting guide http://www.R-project.org/posting-guide.html[http://www.R-project.org/posting-guide.html] and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.