[R] Thread parallelism and memory management on shared-memory supercomputers

I've got allocations on a couple of shared memory supercomputers, which I use to run computationally-intensive scripts on multiple cores of the same node. I've got 24 cores on the one, and 48 on the other. In both cases, there is a hard memory limit, which is shared among the cores in the nod

Re: [R] Two Time Fixed Effects - LFE package

> LFE package? > > felm(outcome ~ temperature + precipitation | city + year + week > > Thanks again! > > Sincerely, > > MS > > On Sun, Nov 15, 2015 at 12:13 AM, Andrew Crane-Droesch > mailto:andre...@gmail.com>> wrote: > > Is this an R question or an eco

Re: [R] Two Time Fixed Effects - LFE package

Is this an R question or an econometrics question? I'll assume that it is an R question. If your weeks are coded sequentially (i.e.: weeks since a particular date), then they'll be strictly determined by year. If however you're interested in the effect of a particular week of the year (week

Re: [R] plot.gam help

Your problem is that you've specified a parametric linear model. Your output is a set of parametric coefficients. You can replace `gam` with `lm` in your example, and you'll get identical results. If you wish to specify one of the terms nonparametrically, you'd use s(chlag), for example (assum

Re: [R] R packages/code for CART that is written entirely in R (not C)

Thanks for the reference! I'd still be grateful for any pointers towards code for CART that has been written entirely in R, perhaps that hasn't made its way onto CRAN. On 11/11/2015 02:00 PM, David Winsemius wrote: On Nov 11, 2015, at 9:25 AM, Andrew Crane-Droesch wrote: Dear

[R] R packages/code for CART that is written entirely in R (not C)

Dear List, I'd like to make a few modifications to the typical CART algorithm, and I'd rather not code the whole thing from scratch. Specifically I want to use different in-sample and out-of-sample fit criteria in the split choosing and cross-validation stages. I see however that the code for

Re: [R] Is there a convenient way of extracting the matrix `solve(X %*% t(X) + PENALTY)` from an additive model fit in mgcv?

> b <- gam(y~s(x0)+s(x1)+s(x2)+s(x3),data=dat) > ## extract the thing required in the title > b$Vp/b$sig2 > > best, > Simon > > > On 18/06/15 03:23, Andrew Crane-Droesch wrote: >> The title says it all. An additive model can be fit by `solve(X %*% t(X) >>

[R] Is there a convenient way of extracting the matrix `solve(X %*% t(X) + PENALTY)` from an additive model fit in mgcv?

The title says it all. An additive model can be fit by `solve(X %*% t(X) + PENALTY)%*%t(X)%*%y` (though of course there are more efficient ways to do it). I want the matrix `solve(X %*% t(X) + PENALTY)` from a fitted gam object. GAM objects can be a bit tricky to navigate -- is there a convenie

Re: [R] bam (mgcv) not using the specified number of cores

nformation, please. In particular the results of sessionInfo() and the code that caused this weird behaviour (+ an indication of dataset size). best, Simon On 21/08/14 12:53, Andrew Crane-Droesch wrote: I am getting strange behavior when trying to fit models to large datasets using bam. I am

[R] bam (mgcv) not using the specified number of cores

I am getting strange behavior when trying to fit models to large datasets using bam. I am working on a 4-core machine, but I think that there may be 2 physical cores that the computer uses as 4 cores in some sense that I don't understand. When I run the bam using makeCluster(3), the model run

Re: [R] Seeking well-commented versions of mgcv source code

It turns out that commented versions of the source code are available on GitHub: https://github.com/cran/mgcv/blob/master/R/mgcv.r __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http:

[R] Seeking well-commented versions of mgcv source code

Dear List, I'm looking for well-commented versions of various functions comprising mgcv, so that I can modify a piece of it for a project I'm working on. In particular I'm looking for * testStat * summary.gam * liu2 * simf Obviously I can find these by typing mgcv:::whatever. But the

[R] PLM falling into the dummy variable trap -- how to fix?

Dear list, CC Dr Croissant, I'm tryng to use plm with timeXgroup effects (in addition to the twoway effects), and running into problems with dummy variables. Eventually I want to take advantage of pgmm as well, but first I'd like to solve the problem with the non-dynamic version. An example:

[R] PLM falling into dummy variable trap -- how to fix?

**Appologies for cross-posting with Stack Overflow** An example of the problem: load(url('http://andrewcd.berkeley.edu/sdat')) head(sdat) library(plm) fem = plm(y~T+G:t,data=sdat,effect="twoways",model="within",index=c("ID","t")) summary(fem) lsdvm = lm(y~ID+T+G:t,data=

[R] Inscrutable error message in mgcv: 1> prediction = predict(MI, se.fit=TRUE, newdata=rhc), Error in if (object$inter) X[[i]] <- PredictMat(object$margin[[i]], dat, : , argument is of length zero

Dear List, I am trying to reproduce a figure that I made for an analysis that I did a few months ago. Between when I first made the figure and now, I've upgraded to R 3.0.2 and upgraded my operating system (ubuntu 13.04). My codebase, which once works, is throwing an error when I try to use

Re: [R] Instructions for upgrading R on ubuntu

ying Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. Andrew Crane-Droesch wrote: I am

[R] Instructions for upgrading R on ubuntu

I am trying to upgrade to R 3.0.1, and I am working on ubuntu. Apparently, the new version is available in the following ppa: https://launchpad.net/~marutter/+archive/c2d4u. I have added this ppa to my software sources. I then run the typical sudo apt-get update && sudo apt-get dist-upgrade,

[R] Modifying the internals of mgcv's G object to program a within estimator

I'm trying to write a function to make gam (in mgcv) give me the fixed-effects "within" estimator, which is equivalent to the OLS dummy variable estimator. Basically this involves subtracting the within-group means from the IVs and the DV, adding their overall means back in, and fitting the mo

Re: [R] Modifying a design matrix in mgcv

Thanks! I hadn't touched environment variables before, not knowing what they were. The G$X matrix indeed seems to get squared, giving appropriately nonsensical results. I can now go and code a (hopefully) sensible change to it. On 08/14/2013 10:32 PM, David Winsemius wrote: environment(mo

[R] Modifying a design matrix in mgcv

Hello, I am trying to make a slight modification to the way gam (mgcv) works. I want to modify the G$X matrix, which is the design matrix, to accommodate an estimator that I am trying to program. I am working with panel data, and I want to take all continuous variables, including basis funct

[R] gam (mgcv), multiple imputation, f-stats/p-values, and summary(gam)

Dear All, I'm using gam for a project that involves multiple imputation, and it has led me to a question about how f-statistics/p-values work in gam. Specifically, how do the values in summary(gam) get generated? As is made clear by the dumb example below, I'm manipul;ating gam objects to r

Re: [R] Understanding why a GAM can't suppress an intercept

ifiability. > > There is a solution to forcing a spline through a particular point at > http://r.789695.n4.nabble.com/Use-pcls-in-quot-mgcv-quot-package-to-achieve-constrained-cubic-spline-td4660966.html > > > (i.e. the R help thread "Re: [R] Use pcls in "mgcv" package to a

Re: [R] Understanding why a GAM can't have an intercept

please deleter this thread -- wrong title On 04/16/2013 02:35 PM, Andrew Crane-Droesch wrote: > Dear List, > > I've just tried to specify a GAM without an intercept -- I've got one > of the (rare) cases where it is appropriate for E(y) -> 0 as X ->0. > Nai

[R] Understanding why a GAM can't have an intercept

Dear List, I've just tried to specify a GAM without an intercept -- I've got one of the (rare) cases where it is appropriate for E(y) -> 0 as X ->0. Naively running a GAM with the "-1" appended to the formula and the calling "predict.gam", I see that the model isn't behaving as expected. I do

[R] Understanding why a GAM can't suppress an intercept

> Dear List, > > I've just tried to specify a GAM without an intercept -- I've got one > of the (rare) cases where it is appropriate for E(y) -> 0 as X ->0. > Naively running a GAM with the "-1" appended to the formula and the > calling "predict.gam", I see that the model isn't behaving as exp

Re: [R] [mgcv][gam] Odd error: Error in PredictMat(object$smooth[[k]], data) : , `by' variable must be same dimension as smooth arguments

Solved my own problem. If interested: http://stackoverflow.com/questions/15563589/odd-error-error-in-predictmatobjectsmoothk-data-by-variable-must On 03/21/2013 02:14 PM, Andrew Crane-Droesch wrote: Dear List, I'm getting an error in mgcv, and I can't figure out where it comes

[R] [mgcv][gam] Odd error: Error in PredictMat(object$smooth[[k]], data) : , `by' variable must be same dimension as smooth arguments

Dear List, I'm getting an error in mgcv, and I can't figure out where it comes from. The setup is the following: I've got a fitted GAM object called "MI", and a vector of "prediction data" (with default values for predictors). I feed this into predict.gam(object, newdata = whatever) via th

Re: [R] How to automate this model selection algorithm?

Just to add, I've been playing around with "select=TRUE" in mgcv, and it does seem that it could work if I were to specify all of the nested two-way interactions in my three-way interactions (see the toy example below). But the problem is that I don't have enough degrees of freedom to feed suc

[R] How to automate this model selection algorithm?

I've got a complicated semi-parametric model that I'm fitting with mgcv. I start with a model based on theory. Its got lots of interaction terms. I want to winnow it down: removing each interaction term or un-interacted main effect one by one, checking the AIC, and retaining the model that g

[R] R code parallelized using plyr and doMC: error message: Error in do.ply(i) : task 1 failed - “could not find function ”getClass“”

Dear list, I'm just getting started learning how to use remote supercomputers for execution of parallelized code. I got a lot of initial help from this previous post, as wel

Re: [R] Why are the number of coefficients varying? [mgcv][gam]

Hi Simon, Thanks for replying. On further investigation, I can't reproduce this error on my local machine -- it only occurs when sending to a cluster (to run the multiple imputations in parallel) that I've got access to. I send to a friend's web server, and I get the same sort of error (but

[R] Why are the number of coefficients varying? [mgcv][gam]

EML" ,weights = 1/nsame ), error=function(e) e, finally=doyee) if(inherits(r.ints, "error")) {r.ints=doyee; print("an error happened but it got handled.")} -- *Andrew Crane-Droesch* Energy and Resources Group UC Berkeley +1 215 435 2644 andre...@berkeley.edu sky

[R] [mgcv][gam] Problem defining axis labels for non-smooth terms (via termplot)

Dear List, plot.gam appears to be having trouble communicating its xlab and ylab information to termplot. A simple example: library(mgcv) x = 1:1000 y = runif(1000)*x^.5 z = rnorm(1000)*y other = sin(z) fit = gam(y~s(x)+s(z)+other) plot(fit,all.terms=TRUE) plot(fit,select=3,xlab="???") de

[R] How to get transparent colors to sum to complete opacity?

Dear List, I want to use transparency in R to represent downweighting of observations based on clusters (repeated observations in a dataset). Some clusters will have identical covariate values in a parameter space -- in the 2D x,y case, these represent a bunch of semi-tranparent dots in the

Re: [R] Multiple palettes on single plot don't get rendered when I use dev.copy2pdf

is: pdf("broke.pdf") palette(rainbow(6)) plot(x=x1,y=y1,col=y1,xlim=c(-10,20)) palette(heat.colors(6)) points(x=x2,y=y2,col=y2) dev.off() A.K. - Original Message ----- From: Andrew Crane-Droesch To: R help Cc: Sent: Tuesday, December 11, 2012 7:17 PM Subject: [R] Multiple pal

[R] Multiple palettes on single plot don't get rendered when I use dev.copy2pdf

Hi All, I'm having trouble with the colors on my screen getting translated to the colors in the outputted .pdf document. Here is a caricature of my problem: par(mfrow=c(1,1)) x1 = rnorm(1000) x2 = rnorm(1000)+10 y1 = rnorm(1000)+10 y2 = rnorm(1000)+10 palette(rainbow(6)) plot(x=x1,y=y1,col=y1

Re: [R] [mgcv][gam] Manually defining my own knots?

dumb.example = gam(y~s(x,k=3),knots=list(x=dumb.knots)) best, Simon On 29/11/12 23:44, Andrew Crane-Droesch wrote: Dear List, I'm using GAMs in a multiple imputation project, and I want to be able to combine the parameter estimates and covariance matrices from each completed dataset'

[R] [mgcv][gam] Manually defining my own knots?

Dear List, I'm using GAMs in a multiple imputation project, and I want to be able to combine the parameter estimates and covariance matrices from each completed dataset's fitted model in the end. In order to do this, I need the knots to be uniform for each model with partially-imputed data.

[R] [gam] [mgcv] Question in integrating a eiker-white "sandwich" VCV estimator into GAM

#x27;d like to be able to make a graph that adds the estimate of the coefficient on T to the smooth function of T*var2, and also adds their standard errors. Of course, I'd like these to be the standard errors estimated by the "sandwich" estimator, above. Thanks for any help an

Re: [R] Help with programming a tricky algorithm

sub[, c("name", "idxy", "ix", iy")] dput( head(exmpl, 30) ) # paste the output of this in a post And also try to change the column name like I've said above. But a data example would really be needed. Hope this helps, Rui Barradas Em 22-10-2012 16:26, And

Re: [R] Help with programming a tricky algorithm

Rui, Thanks a lot for your help. Unfortunately this doesn't work though: 1> is.border <- function(idx, DF){ 1+ i1 <- DF$ix %in% (DF$ix[idx] + c(-1, 1)) & DF$iy == DF$iy[idx] 1+ i2 <- DF$iy %in% (DF$iy[idx] + c(-1, 1)) & DF$ix == DF$ix[idx] 1+ any(DF$country[idx] != DF$country[i1 | i

[R] Help with programming a tricky algorithm

Hi All, I'm a little stumped by the following problem. I've got a dataset with the following structure: idxyixiycountry(other variables) 111c1x1 212c1x2 313c1x3 ... .

[R] gam (mgcv) problem: Error in while (mean(ldxx/(ldxx + ldss)) > 0.4) { :, missing value where TRUE/FALSE needed

Hi All, I'm running into a problem with GAM (in the MGCV package). When I try to estimate the model, I get the following error message: 1> fit <- gam(ndvi~s(rain)+s(temp)+s(rainl1)+s(rainl2)+s(rainxY)+s(rainl1xY)+s(rainl2xY)+s(tempxY), data=dsub, weights=wvec) Error in while (m

[R] SPM/SemiPar -- Plotting additive interactions

I'm taking the residual-regression approach to semiparametric estimation (Robinson 1988, Econometrica), and basically using SemiPar simply as a convenient means of doing multivariate nonparamteric additive models. The final bit of code is here: finalfit <- spm(res~f(V3,basis="trunc.poly")+f(V5