Re: [R] Collapsing data.frame to its or xts
On Wed, Jun 4, 2014 at 3:52 PM, Costas Vorlow costas.vor...@gmail.com wrote:
Hello,
I have a data.frame of a time series sampled every 15 minutes. Also the
data are in reverse order (recent to older). It looks like the following:
snip lots of data
Each row has 96 elements (excluding the date column), i.e., there are 96
columns of 15 minute intervals from 0:00 time to 23:45 apart from the 1st
column that has the dates in dd/mm/yy format. The data start on the 30th of
June, 2011, 0:00.
I made a rudimentary function to read the data.frame from its csv file,
reverse it and create a single time series (xts) that has a 15min frequency.
preparehfdata-function(x,ISOdate=ISOdatetime(2011,06,29,23,45,0))
{
mydata-x[nrow(x):1,]
temp-(as.matrix(mydata))
temp2-temp[,-1]
temp3-as.numeric(apply(temp2, MARGIN=1, FUN=as.vector))
hoursminutes - ISOdate + seq(0:(length(temp3)-1))*15*60
temp4-xts(temp3, order.by=hoursminutes)
return(temp4)
}
which works fine except from the part that the data.frame is missing some
days and my time alignment overshoots. I.e., although there should have
been observations (96 15-minute prices) for every day since June 30th 2011,
some days are missing and some days are repeated (see rows 3, 4 and 5 for
example above) in the data.frame, with the same prices...
Is there an easier way to omit copies of same date data and account for the
missing days when collapsing the dataframe to a single time series (xts or
zoo)?
Preferably I would like to avoid treating this as an irrelevantly spaced
time series (its) if possible...
Many thanks in advance for your input,
Best,
Costas
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
It will be a lot easier, and therefore more likely, for others to help
you if you follow the instructions in the footer. Here are some
suggestions of how to create a *minimal*, reproducible example:
http://stackoverflow.com/q/5963269/271616
Best,
--
Joshua Ulrich | about.me/joshuaulrich
FOSS Trading | www.fosstrading.com
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect escape character
You'll find it even more entertaining to realize that Frede's _previous_ position (at Department of Genetics and Biotechnology, Faculty of Agricultural Sciences, Aarhus University) was quite a bit closer to yours. ;-) -pd On 03 Jun 2014, at 20:37 , Sarah Goslee sarah.gos...@gmail.com wrote: Frede Aakmann Tøgersen Specialist, M.Sc., Ph.D. Plant Performance Modeling I find this very entertaining, because that's exactly what I do -- Plant Performance and Modeling -- but when I do it it involves botany. :) Sarah -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] access scopus data
Hello, The Scopus bibliographic database allows one to manually download publications. Is there any R package for accessing scopus data ? How can it be accessed in R? Thanks: John __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Trucated Normal Function
Dear all members
The following function is truncated normal distribution function i wrote it
by using winbugs so now i want to write it by using R - program.
for(j in
1:P){y[i,j]~djl.dnorm.trunc(mu[i,j],psi[j],thd[j,z[i,j]],thd[j,z[i,j]+1])}
where y:underlying continuous variables and treated as a missing variable
and augmented wit the posterior analysis.
mu: mean
psi: variance
z:ordered categorical variables
the : preassigned values for thresholds and represent matrix(10*6)
dimention.
I hope anyone help me
Many thanks in advance
Thanoon
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect escape character
Yes, and at that time there were in fact some people in the department studying biodiversity or perhaps I should say the lack of biodiversity in the modern agricultural farmland in Denmark. And now some years later things are sadly not better. Today the plants I'm studying is wind power plants, better than nuclear power plants :-) Best regards Frede Sendt fra Samsung mobil Oprindelig meddelelse Fra: peter dalgaard Dato:05/06/2014 13.15 (GMT+01:00) Til: Sarah Goslee Cc: Frede Aakmann Tøgersen ,r-help@r-project.org,Adrian DuÈa Emne: Re: [R] detect escape character You'll find it even more entertaining to realize that Frede's _previous_ position (at Department of Genetics and Biotechnology, Faculty of Agricultural Sciences, Aarhus University) was quite a bit closer to yours. ;-) -pd On 03 Jun 2014, at 20:37 , Sarah Goslee sarah.gos...@gmail.com wrote: Frede Aakmann Tøgersen Specialist, M.Sc., Ph.D. Plant Performance Modeling I find this very entertaining, because that's exactly what I do -- Plant Performance and Modeling -- but when I do it it involves botany. :) Sarah -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] distance by sea
Hello, someone know if it is possible to use R to calculate distance by sea between two geographic coordinates? I have many points in the sea and I want to create a matrix using R of the length of trajectories that pass only trough sea. Thanks Giulia -- Giulia Fassio Animal Biology PhD student Dept. of Biology and Biotechnology C. Darwin Sapienza University of Rome Piazzale Aldo Moro 5, 00185 Rome, Italy Mobile: +39 328 1214497 giuliafas...@gmail.com [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] big matrix reading and writing
All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] big matrix reading and writing
How are you writing it out now? Are you using 'save' which will compress the file? What are the range of numbers in the matrix? Can you scale them to integers (what is the range of the numbers) which might save some space? You did not provide enough information to make a definitive solution. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] error in R program
Hello again Thanoon,
Once again, you should send these request not to me but to the r-help
list. You are far more likely to get help from the greater R community
than just me. Furthermore, it is not entirely clear where your error is.
It is courteous to provide only the code that is run up to the error and
commenting the location. I see you have a loop and all the more important
to isolate where the error is taking place and commenting it out.
My recommendations:
1. Set both t and i = 1 and try to run your loop code manually to locate
the specific function providing the error.
2. Check your data inputs for the function. The error tells you that some
data is missing or infinite. Perhaps a slight error in your data
generation?
3. See if you can address the specific instance before running the full
loops, it may be multiple instances or just one.
I hope this provides some guidance,
Charles
On Thu, Jun 5, 2014 at 3:10 AM, thanoon younis thanoon.youni...@gmail.com
wrote:
Dear Dr. Charles
i need your help to correct the winbugs code below to estimate parameters
of SEM by using bayesian inference for two group. I have this errorError
in eigen(Sigma, symmetric = TRUE, EISPACK = EISPACK) : infinite or missing
values in 'x'.
many thanks in advance
R-CODE
library(MASS) #Load the MASS package
library(R2WinBUGS) #Load the R2WinBUGS package
library(boa) #Load the boa package
library(coda) #Load the coda package
N1-200;N2=100; P1-10;P2-10
phi1-matrix(NA,nrow=4,ncol=4) #The covariance matrix of xi1
Ro1-matrix(NA,nrow=4,ncol=4)
yo1-matrix(data=NA,nrow=200,ncol=10)
phi2-matrix(NA,nrow=4,ncol=4) #The covariance matrix of xi2
Ro2-matrix(NA,nrow=4,ncol=4)
yo2-matrix(data=NA,nrow=200,ncol=10)
#Matrices save the Bayesian Estimates and Standard Errors
Eu1-matrix(data=NA,nrow=100,ncol=10)
SEu-matrix(data=NA,nrow=100,ncol=10)
Elam1-matrix(data=NA,nrow=100,ncol=6)
SElam1-matrix(data=NA,nrow=100,ncol=6)
Egam1-matrix(data=NA,nrow=100,ncol=4)
SEgam1-matrix(data=NA,nrow=100,ncol=4)
Ephx1-matrix(data=NA,nrow=100,ncol=4)
SEphx1-matrix(data=NA,nrow=100,ncol=4)
Eb1-numeric(100); SEb-numeric(100)
Esgd1-numeric(100); SEsgd-numeric(100)
Eu2-matrix(data=NA,nrow=100,ncol=10)
SEu2-matrix(data=NA,nrow=100,ncol=10)
Elam2-matrix(data=NA,nrow=100,ncol=6)
SElam2-matrix(data=NA,nrow=100,ncol=6)
Egam2-matrix(data=NA,nrow=100,ncol=3)
SEgam2-matrix(data=NA,nrow=100,ncol=3)
Ephx2-matrix(data=NA,nrow=100,ncol=3)
SEphx2-matrix(data=NA,nrow=100,ncol=3)
Eb2-numeric(100); SEb2-numeric(100)
Esgd2-numeric(100); SEsgd2-numeric(100)
#Arrays save the HPD intervals
mu.y1=array(NA, c(100,10,2))
lam1=array(NA, c(100,6,2))
gam1=array(NA, c(100,4,2))
sgd1=array(NA, c(100,2))
phx1=array(NA, c(100,4,2))
mu.y2=array(NA, c(100,10,2))
lam2=array(NA, c(100,6,2))
gam2=array(NA, c(100,3,2))
sgd2=array(NA, c(100,2))
phx2=array(NA, c(100,3,2))
DIC=numeric(100)#DIC values
#Parameters to be estimated
parameters-c
(mu.y1,lam1,gam1,phi1,psi1,psd1,sgd1,phx1,mu.y2,lam2,gam2,phi2,psi2,psd2,sgd2,phx2)
#Initial values for the MCMC in WinBUGS
init1-list(uby1=rep(0.0,10),lam1=rep(0.0,10),
gam1=c(1.0,1.0,1.0,1.0),psd1=1.0,phi1=matrix(data=c(1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0),ncol=4,byrow=TRUE),psi1=rep(0.0,10),
xi1=matrix(data=rep(0.0,200),ncol=4),xi2=matrix(data=rep(0.0,200),ncol=4))
init2-list(mu.y1=rep(0.0,10),lam1=rep(0.0,10),
gam1=c(1.0,1.0,1.0,1.0),psd1=1.0,phi1=matrix(data=c(1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0),ncol=4,byrow=TRUE),psi1=rep(0.0,10),xi1=matrix(data=rep
(0.0,200),ncol=4),xi2=matrix(data=rep(0.0,200),ncol=4))
inits-list(init1, init2)
#Do simulation for 100 replications
for (t in 1:100) {
#Generate the data for the simulation study
for (i in 1:N1) {
#Generate xi1
xi1-mvrnorm(1,c(0,0,0,0),phi1, tol = 1e-6, empirical = FALSE,
EISPACK = FALSE)
#Generate error term is structural equation
delta-rnorm(1,0,sqrt(0.3))
#Generate eta1 according to the structural equation
eta1-xi1[1]+xi1[2]+xi1[3]+xi1[1]*xi1[2]+delta
#Generate error terms in measurement equations
eps1-rnorm(3,0,1)
#Generate theta according to measurement equations
Y1[i,1]=Xi1[i,1]+eps1[1]
Y1[i,2]=0.8*Xi1[i,1]+eps1[2]
Y1[i,3]=0.8*Xi1[i,1]+eps1[3]
Y1[i,4]=0.8*Xi1[i,1]+eps1[4]
Y1[i,5]=Xi1[i,2]+eps1[5]
Y1[i,6]=0.8*Xi1[i,2]+eps1[6]
Y1[i,7]=Xi1[i,3]+eps1[7]
Y1[i,8]=Xi1[i,2]+eps1[8]
Y1[i,9]=Eta1[i]+eps1[9]
Y1[i,10]=0.8*Eta1[i]+eps1[10]
}
#transform theta to orinal variables
for (j in 1:10) { if (Y1[j]0) yo1[i,j]-1 else yo1[i,j]-0 }
#Input data set for WinBUGS
data-list(N1=200,N2=200,P=10,R=Ro,z=yo1)
#Call WinBUGS
model-bugs(data,inits,parameters,model.file=D:/Run/model.txt,
Re: [R] big matrix reading and writing
Jim At the moment I'm using write.table. I tried using write.matrix from the MASS package, but that failed. Integers are not appropriate here because we are working with fractions of miles for some locations and that needs to be retained. The range is from 0 to about 3.5 (it's a little less than that with the digits) I haven't tried the save function yet, but I wasn't aware of that one previously. Thanks for pointing that out. The bigger concern is reading and working with that dataset in the other calculation though. Adrienne On Thu, Jun 5, 2014 at 9:37 AM, jim holtman jholt...@gmail.com wrote: How are you writing it out now? Are you using 'save' which will compress the file? What are the range of numbers in the matrix? Can you scale them to integers (what is the range of the numbers) which might save some space? You did not provide enough information to make a definitive solution. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] big matrix reading and writing
The real question is how much memory does the machine that you are working on have. The 32000x32000 matrix will take up ~8GB of physical memory, so how much memory will the rest of your objects take up. Are any of them going to be copies of the distance matrix, or is it always going to be unchanged? Normally my rule of thumb is that I should have 3-4 times the largest object I am working with since I may be making copies of it. So it is important to understand how the rest of your program will be using this large matrix and what type of operations you will be doing on it. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:48 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: Jim At the moment I'm using write.table. I tried using write.matrix from the MASS package, but that failed. Integers are not appropriate here because we are working with fractions of miles for some locations and that needs to be retained. The range is from 0 to about 3.5 (it's a little less than that with the digits) I haven't tried the save function yet, but I wasn't aware of that one previously. Thanks for pointing that out. The bigger concern is reading and working with that dataset in the other calculation though. Adrienne On Thu, Jun 5, 2014 at 9:37 AM, jim holtman jholt...@gmail.com wrote: How are you writing it out now? Are you using 'save' which will compress the file? What are the range of numbers in the matrix? Can you scale them to integers (what is the range of the numbers) which might save some space? You did not provide enough information to make a definitive solution. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] distance by sea
If it is possible at all (and I suspect it is) then it should be possible in R. However, your question is lazy... you should not be asking about black boxes that magically solve your problem, but rather do a search for algorithms that accomplish your goals (wouldn't you want to know if someone had found a solution that handled staying on land?), and then search for (using Google or RSiteSearch or package sos) implementations of the algorithm. At least that way you would encounter options to help do it yourself or use a different tool for that task if no canned solution were available. Also, the R-sig-geo list would be more likely to have expertise in this area, especially if you described some algorithms you had already looked at. --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On June 5, 2014 5:58:11 AM PDT, Giulia Fassio giuliafas...@gmail.com wrote: Hello, someone know if it is possible to use R to calculate distance by sea between two geographic coordinates? I have many points in the sea and I want to create a matrix using R of the length of trajectories that pass only trough sea. Thanks Giulia __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] big matrix reading and writing
Jim, There are not going to be additional copies of the distance matrix. The distance matrix is what is needed for a geographically weighted regression, so I can estimate the results from that to be a SpatialPointsDataFrame at roughly 3 rows by 7 columns. Much smaller size than the distance matrix, though I'm not sure given the class of that object. Potentially lots of room taken up, at the moment the machine in question has 32GB to work with, but we may be able to shift this onto machines with more RAM. A On Thu, Jun 5, 2014 at 10:00 AM, jim holtman jholt...@gmail.com wrote: The real question is how much memory does the machine that you are working on have. The 32000x32000 matrix will take up ~8GB of physical memory, so how much memory will the rest of your objects take up. Are any of them going to be copies of the distance matrix, or is it always going to be unchanged? Normally my rule of thumb is that I should have 3-4 times the largest object I am working with since I may be making copies of it. So it is important to understand how the rest of your program will be using this large matrix and what type of operations you will be doing on it. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:48 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: Jim At the moment I'm using write.table. I tried using write.matrix from the MASS package, but that failed. Integers are not appropriate here because we are working with fractions of miles for some locations and that needs to be retained. The range is from 0 to about 3.5 (it's a little less than that with the digits) I haven't tried the save function yet, but I wasn't aware of that one previously. Thanks for pointing that out. The bigger concern is reading and working with that dataset in the other calculation though. Adrienne On Thu, Jun 5, 2014 at 9:37 AM, jim holtman jholt...@gmail.com wrote: How are you writing it out now? Are you using 'save' which will compress the file? What are the range of numbers in the matrix? Can you scale them to integers (what is the range of the numbers) which might save some space? You did not provide enough information to make a definitive solution. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] distance by sea
Package geosphere has functions to compute the great circle distance between any two points given the latitude and longitude. It does not care if they are at sea, but does not take topography into account. - David L Carlson Department of Anthropology Texas AM University College Station, TX 77840-4352 -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Jeff Newmiller Sent: Thursday, June 5, 2014 9:16 AM To: Giulia Fassio; r-help@r-project.org Subject: Re: [R] distance by sea If it is possible at all (and I suspect it is) then it should be possible in R. However, your question is lazy... you should not be asking about black boxes that magically solve your problem, but rather do a search for algorithms that accomplish your goals (wouldn't you want to know if someone had found a solution that handled staying on land?), and then search for (using Google or RSiteSearch or package sos) implementations of the algorithm. At least that way you would encounter options to help do it yourself or use a different tool for that task if no canned solution were available. Also, the R-sig-geo list would be more likely to have expertise in this area, especially if you described some algorithms you had already looked at. --- Jeff NewmillerThe . . Go Live... DCN:jdnew...@dcn.davis.ca.usBasics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/BatteriesO.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --- Sent from my phone. Please excuse my brevity. On June 5, 2014 5:58:11 AM PDT, Giulia Fassio giuliafas...@gmail.com wrote: Hello, someone know if it is possible to use R to calculate distance by sea between two geographic coordinates? I have many points in the sea and I want to create a matrix using R of the length of trajectories that pass only trough sea. Thanks Giulia __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] big matrix reading and writing
Adrienne, I have a crew just starting work on this problem this week, but I think in your case the best solution is more memory. R stores the distance matrix as a vector with (n^2-n)/2 entries. It's perfectly dense and immune to sparse matrix compaction. It would be quite possible to multiply your values by 1000 and store as integers (given your range of values) and reduce the space required significantly, but the regression function you ultimately pass this to expects doubles and you would have to cast the values back to double on the fly, resulting in wasted time and a matrix that is as big as it would have been anyway. Our view is that while it's possible to store the distance matrix on disk rather than memory, all the functions that accept this matrix as an argument also have to re-written to work with the distances on disk. We're looking into doing this for PCO and NMDS, but someone would have to do the same for the geographically weighted regression. I'm sure that's doable, but certainly not trivial. Dave On 06/05/2014 08:56 AM, Adrienne Wootten wrote: Jim, There are not going to be additional copies of the distance matrix. The distance matrix is what is needed for a geographically weighted regression, so I can estimate the results from that to be a SpatialPointsDataFrame at roughly 3 rows by 7 columns. Much smaller size than the distance matrix, though I'm not sure given the class of that object. Potentially lots of room taken up, at the moment the machine in question has 32GB to work with, but we may be able to shift this onto machines with more RAM. A On Thu, Jun 5, 2014 at 10:00 AM, jim holtman jholt...@gmail.com wrote: The real question is how much memory does the machine that you are working on have. The 32000x32000 matrix will take up ~8GB of physical memory, so how much memory will the rest of your objects take up. Are any of them going to be copies of the distance matrix, or is it always going to be unchanged? Normally my rule of thumb is that I should have 3-4 times the largest object I am working with since I may be making copies of it. So it is important to understand how the rest of your program will be using this large matrix and what type of operations you will be doing on it. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:48 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: Jim At the moment I'm using write.table. I tried using write.matrix from the MASS package, but that failed. Integers are not appropriate here because we are working with fractions of miles for some locations and that needs to be retained. The range is from 0 to about 3.5 (it's a little less than that with the digits) I haven't tried the save function yet, but I wasn't aware of that one previously. Thanks for pointing that out. The bigger concern is reading and working with that dataset in the other calculation though. Adrienne On Thu, Jun 5, 2014 at 9:37 AM, jim holtman jholt...@gmail.com wrote: How are you writing it out now? Are you using 'save' which will compress the file? What are the range of numbers in the matrix? Can you scale them to integers (what is the range of the numbers) which might save some space? You did not provide enough information to make a definitive solution. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Thu, Jun 5, 2014 at 9:26 AM, Adrienne Wootten amwoo...@ncsu.edu wrote: All, Got a tricky situation and unfortunately because it's a big file I can't exactly provide an example, so I'll describe this as best I can for everyone. I have a distance matrix that we are using for a modeling calculation in space for multiple days. Since the matrix is never going to change for different dates, I want to keep the matrix in a file and refer to that so I don't have to repeat the calculation over and over again for that. The problem is it's a 32000 X 32000 matrix and roughly works out to 15GB of storage. This makes it a trick to read the file back into R, but it leaves me with two questions for the group. Is there anyway to have R write this out so that it takes up less space? I know R primarily treats numbers as doubles, but I'm trying to find a way to get R to write the values as floats or singles. with how big it is, it may not be wise to save it as an object in R when read in, so I'm wondering is there anyway to have R do the calculation it needs to do without saving the matrix as an object in R? Basically can I have it run the calculation off the file itself? Thanks! Adrienne -- Adrienne Wootten Graduate Research Assistant State Climate Office of North Carolina Department of Marine, Earth and Atmospheric Sciences North Carolina State University [[alternative HTML version deleted]]
Re: [R] Trucated Normal Function
library(sos)
tn - findFn('truncated normal', 999)
found 421 matches; retrieving 22 pages
2 3 4 5 6 7 8 9 10
11 12 13 14 15 16 17 18 19 20
21 22
Downloaded 305 links in 138 packages.
tn # opens the list of 305 links in a web browser
installPackages(tn) # installs the packages with the most links
writeFindFn2xls(tn) # creates an Excel file in
getwd() # the working directory
# the first sheet is a package summary
# providing more info for installed packages
# See also
sos - vignette('sos')
sos
NOTES:
1. Many of the 305 links may not reference a truncated
normal, but some of them do.
2. The directory sos$Dir contains doc/sos.R, which contains
the lines of code that appear in sos.pdf, which the print method for
sos above opens in a PDF reader.
Hope this helps.
Spencer
On 6/5/2014 5:10 AM, thanoon younis wrote:
Dear all members
The following function is truncated normal distribution function i wrote it
by using winbugs so now i want to write it by using R - program.
for(j in
1:P){y[i,j]~djl.dnorm.trunc(mu[i,j],psi[j],thd[j,z[i,j]],thd[j,z[i,j]+1])}
where y:underlying continuous variables and treated as a missing variable
and augmented wit the posterior analysis.
mu: mean
psi: variance
z:ordered categorical variables
the : preassigned values for thresholds and represent matrix(10*6)
dimention.
I hope anyone help me
Many thanks in advance
Thanoon
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] R 3.1.0 interpreting large negative seconds since epoch as LMT?
I'm seeing a weird difference in behavior between R versions 3.0.2 and 3.1.0.
Consider this session from version 3.0.2:
secs = c(-6327530133, -632753013, -63275301, -6327530, -632753, -63275,
-6327, -632, -63)
class(secs) = c('POSIXt', 'POSIXct')
secs
[1] 1769-06-27 11:44:27 EST 1949-12-13 06:16:27 EST 1967-12-30 10:31:39
EST 1969-10-19 14:21:10 EDT
[5] 1969-12-24 11:14:07 EST 1969-12-31 01:25:25 EST 1969-12-31 17:14:33
EST 1969-12-31 18:49:28 EST
[9] 1969-12-31 18:58:57 EST
This looks pretty good but if I run the exact same statements on version 3.1.0,
I get a different timezone for the first time:
secs = c(-6327530133, -632753013, -63275301, -6327530, -632753, -63275,
-6327, -632, -63)
class(secs) = c('POSIXt', 'POSIXct')
secs
[1] 1769-06-27 11:48:25 LMT 1949-12-13 06:16:27 EST 1967-12-30 10:31:39
EST 1969-10-19 14:21:10 EDT
[5] 1969-12-24 11:14:07 EST 1969-12-31 01:25:25 EST 1969-12-31 17:14:33
EST 1969-12-31 18:49:28 EST
[9] 1969-12-31 18:58:57 EST
I've never even heard of LMT before!! I found a page on Wikipedia
(http://en.wikipedia.org/wiki/Local_mean_time) that talks about local mean
time which was used in the 19th century before standard times were adopted.
Is this a bug or a deliberate change in R 3.1.0?
John
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] .Rprofile: contains invalid line(s)
At some point in the recent past, my local .Rprofile has ceased to be
executed on startup. I've upgraded R several times in the last few months, and
am unsure which version caused this problem. Currently I'm running version
3.1.0 (2014-04-10) Spring Dance on 64-bit Ubuntu.
My symptoms and fix attempts:
1) Changes I make to .Rprofile are no longer automatically recognized by R
on startup.
2) After some Googling, I was led to believe that I need to set the
environment variable R_ENVIRON to /home/stephen/.Rprofile. But after doing
so, when I start R, I get an error message: File /home/stephen/.Rprofile
contains invalid line(s). Then it lists the contents of .Rprofile, which are
now simply:
.First - function() {
joe - function(x) x*2
}
This file is perfectly valid, however, which I know because when I
explicitly source() it, it loads .First just fine, and if I run .First(), I
can then successfully run joe(3).
Why does R say this .Rprofile is invalid on startup? Is setting R_ENVIRON,
as described, the correct procedure? And why did this suddenly become
necessary?
- Stephen Davies, Ph.D.
(step...@umw.edu)
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] .Rprofile: contains invalid line(s)
Hi Stephan,
R_ENVIRON should not point to your .Rpofile, use R_PROFILE_USER for
that. See ?Startup for details.
Best,
Ista
On Thu, Jun 5, 2014 at 10:18 AM, Stephen Davies step...@umw.edu wrote:
At some point in the recent past, my local .Rprofile has ceased to be
executed on startup. I've upgraded R several times in the last few months, and
am unsure which version caused this problem. Currently I'm running version
3.1.0 (2014-04-10) Spring Dance on 64-bit Ubuntu.
My symptoms and fix attempts:
1) Changes I make to .Rprofile are no longer automatically recognized by R
on startup.
2) After some Googling, I was led to believe that I need to set the
environment variable R_ENVIRON to /home/stephen/.Rprofile. But after doing
so, when I start R, I get an error message: File /home/stephen/.Rprofile
contains invalid line(s). Then it lists the contents of .Rprofile, which are
now simply:
.First - function() {
joe - function(x) x*2
}
This file is perfectly valid, however, which I know because when I
explicitly source() it, it loads .First just fine, and if I run .First(), I
can then successfully run joe(3).
Why does R say this .Rprofile is invalid on startup? Is setting R_ENVIRON,
as described, the correct procedure? And why did this suddenly become
necessary?
- Stephen Davies, Ph.D.
(step...@umw.edu)
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Reading specific lines from a text file in r
Dear R users,
I am trying to read a text file into R. the text file looks as shown below
(data1). I want to skip the text lines and collect all lines starting
with specific marker(s).
Example in below, I want to collect all lines which start with -1000 and
-1001
If the model results in successful convergence, each run will result in 6
lines per table. I was trying to read the 3rd and 4th line from every block
of 6 lines using the following.
#
n=2 # example only first 2 runs
patt2=NULL
for(ser1 in unique(1:n))
{ patt1 = c(6*ser1-3, 6*ser1-2)
patt2=c(patt2,patt1)
}
see1= read.delim(data1.txt, header=F)
see2 =as.character(see1[,1])
see3 = see2[patt2]
writeLines(see3, see3)
see4 = read.table(see3, header=T, sep='')
see4
V1 V2 V3 V4
1 -1000 2.0 2.0 2.0
2 -1001 0.5 0.5 0.5
3 -1000 2.0 2.0 5.2
4 -1001 0.4 0.4 0.5
##
But then when convergence fails there are only 4 lines generated (see
table 3) for that run. In the model with 1000 or more runs, we donât know
which run will fail to converge. Hence using the pattern of 6 line blocks
doesnât work. Is there a way to directly just read all the lines starting
with -1000 and -1001 ?
Thank you for your time
##data1
table:1 conditional estimation
iteration par1 par2 par3
-1000 2 2 2
-1001 0.5 0.5 0.5
-1002 0 0 0
-1003 0 0 0
table:2 conditional estimation
iteration par1 par2 par3
-1000 2 2 5.2
-1001 0.4 0.4 0.5
-1002 0 0 0
-1003 0 0 0
table:3 conditional estimation
iteration par1 par2 par3
-1000 3.1 2.2 4.2
-1001 0.5 0.5 0.5
table:4 conditional estimation
iteration par1 par2 par3
-1000 2 1.2 2
-1001 0.5 0.4 0.5
-1002 0 0 0
-1003 0 0 0
##
--
Navin Goyal
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Bitwise operations on large numbers
Hi I am working on numbers between 0 - 2^32. I want to perform bit-wise shift operations on these numbers. Integer allows me to do bitwise shift operations but number range is limited to 0 - 2^31. Double allows me to store large numbers but I can not perform bitwise shifting. I can not change programming language as it is part of larger project. Is there a way to solve this problem? Perhaps using data types similar to unsigned integer or long integer? Thanks Koustubh Bagade [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Mapping
Hey Folks, Apologies in advance if this has been covered. I'm taking on a simple mapping project. What packages, tutorials and examples would you recommend for a beginning mapper in R? Thanks, Tom [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Run R Code Continuously in the Background of HTML
I would like to run R code continuously in the background of a HTML page. This code will run in a loop that has a random number generator. I would like the different values from the random number generator to be displayed on the HTML page as the loop iterates. Thanks in advance for any help. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Understanding survey design (svydesign)
Hi all, I am trying to understand the output of the svydesign function. My question is, is there documentation on how the probabilities are computer? Or could someone please explain to me how they are? I have this d-svydesign(id=~PSU, strata=~STRATUM,weights=~w, data=s, nest=TRUE) And this output with summary(d) Stratified 1 - level Cluster Sampling design (with replacement) With (783) clusters. svydesign(id = ~PSU, strata = ~STRATUM, weights = ~Sample_Weight, data = subset25k, nest = TRUE) Probabilities: Min. 1st Qu. MedianMean 3rd Qu.Max. 0 0 0 Inf 0 Inf I have read the documentation about the code but it doesn't actually say anything about the probabilities. Thank you in advance. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Mapping
What type of mapping? Do you want to add points to an existing map? Color in polygons of a downloaded map? Create a completely new map? other? our advice to you depends on what you want to do. Also please read the posting guide linked to at the bottom of all emails and don't post in HTML, this is a plain text list. On Thu, Jun 5, 2014 at 2:00 PM, Tom Detzel tom.det...@propublica.org wrote: Hey Folks, Apologies in advance if this has been covered. I'm taking on a simple mapping project. What packages, tutorials and examples would you recommend for a beginning mapper in R? Thanks, Tom [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Gregory (Greg) L. Snow Ph.D. 538...@gmail.com __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] .Rprofile: contains invalid line(s)
Make sure that the last line has a newline at the end, otherwise that
expression will be silently ignored, cf. R-devel thread '[Rd] Last
line in .Rprofile must have newline (PR#4056)' on 2003-09-03
[https://stat.ethz.ch/pipermail/r-devel/2003-September/027455.html].
I'm pretty sure many many people have been bitten by this feature.
Other than that, add a message(.Rprofile...) and the very top and a
message(.Rprofile...done) at the very bottom to confirm that
~/.Rprofile is loaded/evaluated.
/Henrik
On Thu, Jun 5, 2014 at 10:58 AM, Ista Zahn istaz...@gmail.com wrote:
Hi Stephan,
R_ENVIRON should not point to your .Rpofile, use R_PROFILE_USER for
that. See ?Startup for details.
Best,
Ista
On Thu, Jun 5, 2014 at 10:18 AM, Stephen Davies step...@umw.edu wrote:
At some point in the recent past, my local .Rprofile has ceased to be
executed on startup. I've upgraded R several times in the last few months,
and
am unsure which version caused this problem. Currently I'm running version
3.1.0 (2014-04-10) Spring Dance on 64-bit Ubuntu.
My symptoms and fix attempts:
1) Changes I make to .Rprofile are no longer automatically recognized by
R
on startup.
2) After some Googling, I was led to believe that I need to set the
environment variable R_ENVIRON to /home/stephen/.Rprofile. But after doing
so, when I start R, I get an error message: File /home/stephen/.Rprofile
contains invalid line(s). Then it lists the contents of .Rprofile, which are
now simply:
.First - function() {
joe - function(x) x*2
}
This file is perfectly valid, however, which I know because when I
explicitly source() it, it loads .First just fine, and if I run .First(), I
can then successfully run joe(3).
Why does R say this .Rprofile is invalid on startup? Is setting
R_ENVIRON,
as described, the correct procedure? And why did this suddenly become
necessary?
- Stephen Davies, Ph.D.
(step...@umw.edu)
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Understanding svymean
Hello all, I am trying to understand how svymean is calulated with svydesign. I have the code: dstrat1-svydesign(id=~PSU, strata=~STRATUM,weights=~Sample_Weight, data=subset25k, nest=TRUE) and I am using svymean to compute the mean. My main question is how are the sampling probabilities created? The documentation says that each observation is weighted with the inverse of its sampling probability. Is that coming from the weights that were input in the svydesign? Thank you in advance. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Bitwise operations on large numbers
Double allows me to store large numbers but I can not perform bitwise shifting. It is a nuisance that bitwShiftL and bitwShitR don't just shift (and wrap around to negative) for integers so you could use all 32 bits. You could use doubles and multiply (*) and divide (%/%) by 2^n instead of shifting left and right by n bits. That would give you 52 bits. Bitwise 'and' and 'or' with doubles is harder to do efficiently at the R level but you can get it done with the following: shiftL - function(x, n) x * 2^n shiftR - function(x, n) x %/% 2^n or - function(x, y) .bitOp(x, y, bitwOr) and - function(x, y) .bitOp(x, y, bitwAnd) bitSlice - function(x, from, length) shiftR(x, from) %% 2^(from+length) .bitOp - function(x, y, bitwOp) bitwOp(bitSlice(x, 0, 26),bitSlice(y, 0, 26)) + shiftL(bitwOp(bitSlice(x, 26, 26), bitSlice(y, 26, 26)), 26) Bill Dunlap TIBCO Software wdunlap tibco.com On Thu, Jun 5, 2014 at 11:43 AM, Koustubh Bagade kou.geni...@gmail.com wrote: Hi I am working on numbers between 0 - 2^32. I want to perform bit-wise shift operations on these numbers. Integer allows me to do bitwise shift operations but number range is limited to 0 - 2^31. Double allows me to store large numbers but I can not perform bitwise shifting. I can not change programming language as it is part of larger project. Is there a way to solve this problem? Perhaps using data types similar to unsigned integer or long integer? Thanks Koustubh Bagade [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Error Correction Model on Panel Data
I have a panel data set with a non-stationary regressand and a non-stationary regressor. I've used the plm package to do a fixed effects regression of a long-run (i.e. cointegrating) relationship between these non-stationary variables and I've used plm's purtest to do an Engle-Granger test that the residuals of that regression are indeed stationary. Now, I need to run an error correction model (ECM) to identify the dynamics -- the speed at which the regressand converges to the level implied by that cointegrating relationship in response to a shock to the regressor. The plm package does not seem to be capable of estimating an ECM (unless I missed something). Packages like urca seem to offer ECMs but only for pure time series (i.e. they do not handle panel data). I've searched but have not found other options in r. Any suggestions out there? I've come up with 3 options: 1. Try to code the maximization of the appropriate ECM's likelihood 2. Try to transform an ECM into the sort of dynamic gmm model that can be handled by plm (or even by a package not designed for econometrics like nmle, lme4, or lmer) 3. Try to trick urca (or another package capable of estimating an ECM) into respecting the panel structure of the data by inserting na observations into the time series every time the panel rolls over to the next individual? Thanks for any help, Bentley [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Bitwise operations on large numbers
Oops, I got bitSlice wrong, hence and and or won't be right. bitSlice should be: bitSlice - function(x, from, length) shiftR(x, from) %% 2^length Bill Dunlap TIBCO Software wdunlap tibco.com On Thu, Jun 5, 2014 at 3:31 PM, William Dunlap wdun...@tibco.com wrote: Double allows me to store large numbers but I can not perform bitwise shifting. It is a nuisance that bitwShiftL and bitwShitR don't just shift (and wrap around to negative) for integers so you could use all 32 bits. You could use doubles and multiply (*) and divide (%/%) by 2^n instead of shifting left and right by n bits. That would give you 52 bits. Bitwise 'and' and 'or' with doubles is harder to do efficiently at the R level but you can get it done with the following: shiftL - function(x, n) x * 2^n shiftR - function(x, n) x %/% 2^n or - function(x, y) .bitOp(x, y, bitwOr) and - function(x, y) .bitOp(x, y, bitwAnd) bitSlice - function(x, from, length) shiftR(x, from) %% 2^(from+length) .bitOp - function(x, y, bitwOp) bitwOp(bitSlice(x, 0, 26),bitSlice(y, 0, 26)) + shiftL(bitwOp(bitSlice(x, 26, 26), bitSlice(y, 26, 26)), 26) Bill Dunlap TIBCO Software wdunlap tibco.com On Thu, Jun 5, 2014 at 11:43 AM, Koustubh Bagade kou.geni...@gmail.com wrote: Hi I am working on numbers between 0 - 2^32. I want to perform bit-wise shift operations on these numbers. Integer allows me to do bitwise shift operations but number range is limited to 0 - 2^31. Double allows me to store large numbers but I can not perform bitwise shifting. I can not change programming language as it is part of larger project. Is there a way to solve this problem? Perhaps using data types similar to unsigned integer or long integer? Thanks Koustubh Bagade [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Fwd: error in R program
-- Forwarded message -- From: thanoon younis thanoon.youni...@gmail.com Date: Thursday, June 5, 2014 Subject: error in R program To: Charles Determan Jr deter...@umn.edu many thanks to you Dr. Charles Really i have a problem with simulation data in xi and now i have this erro r Error in mvrnorm(1, c(0, 0, 0), phi1, tol = 1e-06, empirical = FALSE, : incompatible arguments Regards On 5 June 2014 16:45, Charles Determan Jr deter...@umn.edu wrote: Hello again Thanoon, Once again, you should send these request not to me but to the r-help list. You are far more likely to get help from the greater R community than just me. Furthermore, it is not entirely clear where your error is. It is courteous to provide only the code that is run up to the error and commenting the location. I see you have a loop and all the more important to isolate where the error is taking place and commenting it out. My recommendations: 1. Set both t and i = 1 and try to run your loop code manually to locate the specific function providing the error. 2. Check your data inputs for the function. The error tells you that some data is missing or infinite. Perhaps a slight error in your data generation? 3. See if you can address the specific instance before running the full loops, it may be multiple instances or just one. I hope this provides some guidance, Charles On Thu, Jun 5, 2014 at 3:10 AM, thanoon younis thanoon.youni...@gmail.com wrote: Dear Dr. Charles i need your help to correct the winbugs code below to estimate parameters of SEM by using bayesian inference for two group. I have this errorError in eigen(Sigma, symmetric = TRUE, EISPACK = EISPACK) : infinite or missing values in 'x'. many thanks in advance R-CODE library(MASS) #Load the MASS package library(R2WinBUGS) #Load the R2WinBUGS package library(boa) #Load the boa package library(coda) #Load the coda package N1-200;N2=100; P1-10;P2-10 phi1-matrix(NA,nrow=4,ncol=4) #The covariance matrix of xi1 Ro1-matrix(NA,nrow=4,ncol=4) yo1-matrix(data=NA,nrow=200,ncol=10) phi2-matrix(NA,nrow=4,ncol=4) #The covariance matrix of xi2 Ro2-matrix(NA,nrow=4,ncol=4) yo2-matrix(data=NA,nrow=200,ncol=10) #Matrices save the Bayesian Estimates and Standard Errors Eu1-matrix(data=NA,nrow=100,ncol=10) SEu-matrix(data=NA,nrow=100,ncol=10) Elam1-matrix(data=NA,nrow=100,ncol=6) SElam1-matrix(data=NA,nrow=100,ncol=6) Egam1-matrix(data=NA,nrow=100,ncol=4) SEgam1-matrix(data=NA,nrow=100,ncol=4) Ephx1-matrix(data=NA,nrow=100,ncol=4) SEphx1-matrix(data=NA,nrow=100,ncol=4) Eb1-numeric(100); SEb-numeric(100) Esgd1-numeric(100); SEsgd-numeric(100) Eu2-matrix(data=NA,nrow=100,ncol=10) SEu2-matrix(data=NA,nrow=100,ncol=10) Elam2-matrix(data=NA,nrow=100,ncol=6) SElam2-matrix(data=NA,nrow=100,ncol=6) Egam2-matrix(data=NA,nrow=100,ncol=3) SEgam2-matrix(data=NA,nrow=100,ncol=3) Ephx2-matrix(data=NA,nrow=100,ncol=3) SEphx2-matrix(data=NA,nrow=100,ncol=3) Eb2-numeric(100); SEb2-numeric(100) Esgd2-numeric(100); SEsgd2-numeric(100) #Arrays save the HPD intervals mu.y1=array(NA, c(100,10,2)) lam1=array(NA, c(100,6,2)) gam1=array(NA, c(100,4,2)) sgd1=array(NA, c(100,2)) phx1=array(NA, c(100,4,2)) mu.y2=array(NA, c(100,10,2)) lam2=array(NA, c(100,6,2)) gam2=array(NA, c(100,3,2)) sgd2=array(NA, c(100,2)) phx2=array(NA, c(100,3,2)) DIC=numeric(100)#DIC values #Parameters to be estimated parameters-c (mu.y1,lam1,gam1,phi1,psi1,psd1,sgd1,phx1,mu.y2,lam2,gam2,phi2,psi2,psd2,sgd2,phx2) #Initial values for the MCMC in WinBUGS init1-list(uby1=rep(0.0,10),lam1=rep(0.0,10), gam1=c(1.0,1.0,1.0 -- Dr. Charles Determan, PhD Integrated Biosciences [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Why have my glmms stopped converging (lme4)
I have four sets of glmms (binomial, logit-linked) which I have run in various incarnations with no problems over the last weeks. All converged, data assumptions checked, reasonable goodness-of-fit (0.75-85). They are based on three different data sets. Today, I wanted to rerun one of them after amending the data set slightly, and I got the following error message: Warning message:In checkConv(attr(opt, derivs), opt$par, ctrl = control$checkConv, : Hessian is numerically singular: parameters are not uniquely determined I tried re-running the model with the older unchanged version of the data on which the model previously converged, and got the same message. I have not change the model specification at all. I then re-ran my other models which use different data sets to see what would happen, and I got the following message for each: Warning message: In checkConv(attr(opt, derivs), opt$par, ctrl = control$checkConv, : Model failed to converge with max|grad| = 0.00846421 (tol = 0.001) I went back to previous versions of the model and older versions of the data sets and Iâm still getting these error messages, but only for mixed models. For models with only fixed effects, they are as before. But the point is that my mixed models were converging before. Any ideas on what is going on? I might be missing something obvious, but it really seems like this came out of nowhere. Further information about my models: I used the following commands: library(lme4) mod- glmer(y ~ x1 + x2 + x3 + x4 + x5 + (1|x6) + (1|x7) +(1|x8) , data = data, family = binomial (logit)) I am looking at the effects of ecological factors on the presence of pathogens in wildlife. Most of my variables are categorical. Fixed factors are reproductive status, month, presence of particular ectoparasites, and a body condition index. Random factors are site, year, and observer (to account for potential observer-biased condition index). Iâm running R 3.1.0 in Mavericks. This is my first time posting in this list, so I hope my question is acceptably formulated. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Understanding svymean
On Fri, Jun 6, 2014 at 9:50 AM, Ryan de Vera ryan.devera...@gmail.com wrote: Hello all, I am trying to understand how svymean is calulated with svydesign. I have the code: dstrat1-svydesign(id=~PSU, strata=~STRATUM,weights=~Sample_Weight, data=subset25k, nest=TRUE) and I am using svymean to compute the mean. My main question is how are the sampling probabilities created? If you supply weights to svydesign(), those weights are used in the mean calculation. If you supply probabilities, those are used to compute the weights. If you supply population sizes, those are used to compute probabilities, which are then used to compute weights. The code works in terms of probabilities because that's fairly standard in textbooks. It makes it easier for me to get the formulas right. -thomas -- Thomas Lumley Professor of Biostatistics University of Auckland __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Understanding survey design (svydesign)
If you have zeros in the supplied sample weights, the display in the summary() function will look like this. Nothing is actually wrong with the calculations, though. If the zero weights are because of restriction to a subpopulation, you will get better-looking summaries if you put the whole design into svydesign() and then use subset() to restrict to the subpopulation. That way R knows what's going on. It won't affect the answers, though. -thomas On Fri, Jun 6, 2014 at 8:13 AM, Ryan de Vera ryan.devera...@gmail.com wrote: Hi all, I am trying to understand the output of the svydesign function. My question is, is there documentation on how the probabilities are computer? Or could someone please explain to me how they are? I have this d-svydesign(id=~PSU, strata=~STRATUM,weights=~w, data=s, nest=TRUE) And this output with summary(d) Stratified 1 - level Cluster Sampling design (with replacement) With (783) clusters. svydesign(id = ~PSU, strata = ~STRATUM, weights = ~Sample_Weight, data = subset25k, nest = TRUE) Probabilities: Min. 1st Qu. MedianMean 3rd Qu.Max. 0 0 0 Inf 0 Inf I have read the documentation about the code but it doesn't actually say anything about the probabilities. Thank you in advance. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Thomas Lumley Professor of Biostatistics University of Auckland __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect escape character
On Tue, Jun 3, 2014 at 8:44 PM, David Winsemius dwinsem...@comcast.net wrote: On Jun 3, 2014, at 11:03 AM, Adrian Dușa wrote: Dear All, I should be knowing this, but not get it right... I'm a little surprised to see you ask this, too, but we each have lacunae in our R knowledge, so I hope this helps: Thanks very much David, yes indeed sometimes even the most basic things can slip out... but I am grateful for this r-help list. Apologies for the late reply, I am attending a conference all the way in Toronto and the difference in time plus the activity here makes it a bit more difficult to catch up. Best wishes, Adrian -- Adrian Dusa University of Bucharest Romanian Social Data Archive 1, Schitu Magureanu Bd. 050025 Bucharest sector 5 Romania Tel.:+40 21 3126618 \ +40 21 3120210 / int.101 Fax: +40 21 3158391 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] How do I do a conditional sum which only looks between certain date criteria
Hi,
The expected output is confusing.
dat1 - read.table(text=date, user, items_bought
2013-01-01, x, 2
2013-01-02, x, 1
2013-01-03, x, 0
2013-01-04, x, 0
2013-01-05, x, 3
2013-01-06, x, 1
2013-01-01, y, 1
2013-01-02, y, 1
2013-01-03, y, 0
2013-01-04, y, 5
2013-01-05, y, 6
2013-01-06, y, 1,sep=,,header=TRUE,stringsAsFactors=FALSE)
##Assuming that the data is ordered by date and no gaps in date
res1 - unsplit(lapply(split(dat1, dat1$user), function(x) {
indx - (seq(nrow(x)) - 1)%/%3
x$cum_items_bought_3_days - ave(x$items_bought, indx, FUN = cumsum)
x
}), dat1$user)
##expected output
res2 - unsplit(lapply(split(dat1, dat1$user), function(x) {
indx - (seq(nrow(x)) - 1)%/%3
x$cum_items_bought_3_days - ave(x$items_bought, indx, FUN = cumsum)
indx2 - seq(0, length(indx), by = 4)
x[indx2, 4] - x[indx2, 4] + indx[indx2]
x
}), dat1$user)
A.K.
Say I have data that looks like
date, user, items_bought
2013-01-01, x, 2
2013-01-02, x, 1
2013-01-03, x, 0
2013-01-04, x, 0
2013-01-05, x, 3
2013-01-06, x, 1
2013-01-01, y, 1
2013-01-02, y, 1
2013-01-03, y, 0
2013-01-04, y, 5
2013-01-05, y, 6
2013-01-06, y, 1
to get the cumulative sum per user per data point I was doing
data.frame(cum_items_bought=unlist(tapply(as.numeric(data$items_bought),
data$user, FUN = cumsum)))
output from this looks like
date, user, items_bought
2013-01-01, x, 2
2013-01-02, x, 3
2013-01-03, x, 3
2013-01-04, x, 3
2013-01-05, x, 6
2013-01-06, x, 7
2013-01-01, y, 1
2013-01-02, y, 2
2013-01-03, y, 2
2013-01-04, y, 7
2013-01-05, y, 13
2013-01-06, y, 14
However I want to restrict my sum to only add up those that happened within 3
days of each row (relative to the user). i.e. the output needs to look like
this:
date, user, cum_items_bought_3_days
2013-01-01, x, 2
2013-01-02, x, 3
2013-01-03, x, 3
2013-01-04, x, 1
2013-01-05, x, 3
2013-01-06, x, 4
2013-01-01, y, 1
2013-01-02, y, 2
2013-01-03, y, 2
2013-01-04, y, 6
2013-01-05, y, 11
2013-01-06, y, 12
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
