Re: [R] diff-ing .rds files
I was envisioning something involving write.csv(), but this is a better
idea, thanks!
On Mon, Nov 23, 2015 at 3:31 PM, William Dunlap wrote:
> Your any-RDS-to-ASCII-converter could be the R function
>toASCIIRDS <- function (fromRDS, toRDS)
>{
> saveRDS(readRDS(fromRDS), file = toRDS, ascii = TRUE, compress =
> FALSE)
>}
> which you can call from Rscript with appropriate input and output file
> names.
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
>
> On Mon, Nov 23, 2015 at 12:18 PM, Will Hopper
> wrote:
> > Thanks for the input Jeff and Burt, and I could have been more clear
> about
> > what I was looking for.
> >
> > You are of course right that there is nothing preventing me from using
> > plain text files, I initially went with compressed binary files because
> > they were, of course, smaller.
> >
> > What I was curious about is if there was any existing
> > program/script/function/tool that converted an rds to plain text, so
> that I
> > could point git at it when creating the change set for a commit. For
> > example, there are programs that convert pdf documents to plain text, and
> > you can configure git to use these programs to convert the PDF's to plain
> > text before comparing the current version with the version in the last
> > commit, and then only add the things that have changes to the new commit.
> > This allows you to the track changes to a pdf file the same way you
> would a
> > text file, and your delta's don't blow up by committing the entire PDF
> file
> > each time you change part of the file.
> >
> > It would be simple enough to write something like this with some
> > combination of a shell script and an R script to do this same thing for
> > .rds files, but I wondered if there was something out there already I was
> > overlooking.
> >
> > I do see the point that I may be trying to compensate for a flawed
> premise,
> > perhaps I should not use a .rds file here if I'm concerned about the repo
> > getting too big as time goes on. I was just hoping to have my cake and
> eat
> > it too.
> >
> > - Will
> >
> > On Mon, Nov 23, 2015 at 1:33 PM, Jeff Newmiller <
> jdnew...@dcn.davis.ca.us>
> > wrote:
> >
> >> You are sending contradictory signals... do you or do you not want plain
> >> text files? You have not indicated what advantage you are getting from
> >> having binary files in your repository. By far the best answer I see to
> >> your dilemma is to save in ASCII format instead of the default binary
> >> format.
> >>
> >> On November 23, 2015 9:36:21 AM PST, Will Hopper >
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I'm posting to see if anyone knows of any existing resources that
> >>> auto-magically converts r objects in saved in .rds files to a plain
> text
> >>> representation, suitable for diffing?
> >>>
> >>> I often save the results of long running calculation as .rds files, and
> >>> since I use git for source control, it would be nice if there were a
> way to
> >>> convert rds files to a plain text representation for diffing, so I
> could
> >>> avoid having large commits full of binary data. Git allows you specify
> a
> >>> programs for binary --> text conversion for any file type, effectively
> >>> teaching git how to diff binary files. If there was something out there
> >>> developed to do this with .rds files, it would really like to know
> about it!
> >>>
> >>> I realize I could save the rds file with ascii=TRUE and
> compress=FALSE, but
> >>> that kind of defeats the point of saving as .rds in the first place.
> >>>
> >>> If there is no tool out that there anyone
> >>> knows of, I don't think it would
> >>> be too hard for me to write something with bash + Rscript to get the
> job
> >>> done, but I'd like to avoid re-inventing the wheel if possible.
> >>>
> >>> Thanks for the help!
> >>>
> >>> [[alternative HTML version deleted]]
> >>>
> >>> --
> >>>
> >>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible
Re: [R] diff-ing .rds files
Thanks for the input Jeff and Burt, and I could have been more clear about what I was looking for. You are of course right that there is nothing preventing me from using plain text files, I initially went with compressed binary files because they were, of course, smaller. What I was curious about is if there was any existing program/script/function/tool that converted an rds to plain text, so that I could point git at it when creating the change set for a commit. For example, there are programs that convert pdf documents to plain text, and you can configure git to use these programs to convert the PDF's to plain text before comparing the current version with the version in the last commit, and then only add the things that have changes to the new commit. This allows you to the track changes to a pdf file the same way you would a text file, and your delta's don't blow up by committing the entire PDF file each time you change part of the file. It would be simple enough to write something like this with some combination of a shell script and an R script to do this same thing for .rds files, but I wondered if there was something out there already I was overlooking. I do see the point that I may be trying to compensate for a flawed premise, perhaps I should not use a .rds file here if I'm concerned about the repo getting too big as time goes on. I was just hoping to have my cake and eat it too. - Will On Mon, Nov 23, 2015 at 1:33 PM, Jeff Newmiller wrote: > You are sending contradictory signals... do you or do you not want plain > text files? You have not indicated what advantage you are getting from > having binary files in your repository. By far the best answer I see to > your dilemma is to save in ASCII format instead of the default binary > format. > > On November 23, 2015 9:36:21 AM PST, Will Hopper > wrote: > >> Hi all, >> >> I'm posting to see if anyone knows of any existing resources that >> auto-magically converts r objects in saved in .rds files to a plain text >> representation, suitable for diffing? >> >> I often save the results of long running calculation as .rds files, and >> since I use git for source control, it would be nice if there were a way to >> convert rds files to a plain text representation for diffing, so I could >> avoid having large commits full of binary data. Git allows you specify a >> programs for binary --> text conversion for any file type, effectively >> teaching git how to diff binary files. If there was something out there >> developed to do this with .rds files, it would really like to know about it! >> >> I realize I could save the rds file with ascii=TRUE and compress=FALSE, but >> that kind of defeats the point of saving as .rds in the first place. >> >> If there is no tool out that there anyone >> knows of, I don't think it would >> be too hard for me to write something with bash + Rscript to get the job >> done, but I'd like to avoid re-inventing the wheel if possible. >> >> Thanks for the help! >> >> [[alternative HTML version deleted]] >> >> -- >> >> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] diff-ing .rds files
Hi all, I'm posting to see if anyone knows of any existing resources that auto-magically converts r objects in saved in .rds files to a plain text representation, suitable for diffing? I often save the results of long running calculation as .rds files, and since I use git for source control, it would be nice if there were a way to convert rds files to a plain text representation for diffing, so I could avoid having large commits full of binary data. Git allows you specify a programs for binary --> text conversion for any file type, effectively teaching git how to diff binary files. If there was something out there developed to do this with .rds files, it would really like to know about it! I realize I could save the rds file with ascii=TRUE and compress=FALSE, but that kind of defeats the point of saving as .rds in the first place. If there is no tool out that there anyone knows of, I don't think it would be too hard for me to write something with bash + Rscript to get the job done, but I'd like to avoid re-inventing the wheel if possible. Thanks for the help! [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Drop in a Loop
The data structure returned by optimx (that you're storing as gmmiv) is
like a data frame, and stores the conversion code in the field convcode.
So do something like this:
gmmiv =Optimx()
if (gmmiv$convcode ==0) {
store[j,] = coef(gmmiv)
}
On Tue, Sep 15, 2015 at 3:12 PM, Olu Ola wrote:
> Thanks Will.
> Below is the flow of my code
>
> Yhat is the fitted value
> Errhat is the difference between the dependent variable and the yhat
> gmmdata is the data name
>
> N <- nrow(gmmdata)
> B <- 1000
> store <- matrix(0,B,11)
> for (j in 1:B) {
> index = sample(1:N, N, replace=T)
> errnew = errhat[index]
> yt = yhat + errnew
> objective function subroutine
> gradient function subroutine
> gmmiv =Optimx()
> store[j,] = coef(gmmiv)
> }
>
> What I want to do is that if the convergence code from optimx for a
> particular iteration is Not zero, then it should not be stored in store[j,].
>
> Any help will be appreciated
>
> Thank you
>
>
>
>
>
>
>
> On Tue, 9/15/15, Will Hopper wrote:
>
> Subject: Re: [R] Drop in a Loop
> To: "Olu Ola"
> Cc: r-help@r-project.org
> Date: Tuesday, September 15, 2015, 2:30 PM
>
> I
> think you ought to show a small example of how the code
> you're using. Are you saving results at every iteration?
> In a list, data frame, etc? People likely need that to help
> answer your question.
>
> Also probably have a look the control list
> argument and the save.failures option, that might be
> something you're interested in.
>
> - Will
>
> On Tue, Sep 15, 2015 at
> 1:34 PM, Olu Ola via R-help
> wrote:
> Hello,
>
> I am doing some estimation using optimx and after each round
> of estimation, I store the coefficient. However, I need to
> drop the set of coefficients for which the convergence code
> in optimx is GREATER than Zero. How do I go about this?
>
>
>
> A way forward will be highly appreciated.
>
>
>
> Thank you
>
>
>
> __
>
> R-help@r-project.org
> mailing list -- To UNSUBSCRIBE and more, see
>
> https://stat.ethz.ch/mailman/listinfo/r-help
>
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
>
> and provide commented, minimal, self-contained, reproducible
> code.
>
>
>
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Drop in a Loop
I think you ought to show a small example of how the code you're using. Are you saving results at every iteration? In a list, data frame, etc? People likely need that to help answer your question. Also probably have a look the control list argument and the save.failures option, that might be something you're interested in. - Will On Tue, Sep 15, 2015 at 1:34 PM, Olu Ola via R-help wrote: > Hello, > I am doing some estimation using optimx and after each round of > estimation, I store the coefficient. However, I need to drop the set of > coefficients for which the convergence code in optimx is GREATER than Zero. > How do I go about this? > > A way forward will be highly appreciated. > > Thank you > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] optimx parameter scale warning
Hi all, I'm using optimx (version 2013.8.7) to perform parameter estimation with the nelder-mead method, and received a warning that the parameters are on different scales, which can hurt optimization performance on derivative free methods. This warning is accurate as some parameters are small (between 0 and 1) and others can be quite large. So, I tried using the parscale option in the control list to put them on more equal scaling. There are 8 parameters, so I supplied a vector of length 8 to the parscale option, which if I understand correctly, would scale the large parameters down by their starting values so everything will start out near 1 Here is the call to optimx itself I used: fit <- optimx(par = c(ER=.6,LR=.035,TR =.05,FR=.1,alpha=50,lambda=.4,Tmin=.5,Tmax=40), fn = RT_ErrorFcn, method = "Nelder-Mead" control = list(maxit=1000, parscale = c(1,1,1,1,50, 1,1,40)), fcn = model$fn, # passed to RT_ErrorFcn fix = model$fixed, #RT_ErrorFcn obs = data) # passed to RT_ErrorFcn However, the warning about different parameter scales is still given. Is it intended behavior for the warning to be given even when the parscale option is used? Or am I misunderstanding the point of the parscale option, or implementing it wrong? Thanks for any info on this topic. - Will [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
