Re: [Rd] using svd in regression with arima

[Prof Brian Ripley]

On Tue, 6 Jul 2010, Hodgess, Erin wrote:


Dear R Developers:

Why is it that the singular value decomposition is used when running 
regression with arima, please?  I've been looking for a reference 
for that but have come up empty so far.


Do you mean this:

if (!orig.xreg) {
S <- svd(na.omit(xreg))
xreg <- xreg %*% S$v
}

?  That is a stability measure for the numerical optimization (it 
orthogonalizes the columns, ignoring NAs).




Thank you for any help.

Sincerely,
Erin


Erin M. Hodgess, PhD
Associate Professor
Department of Computer and Mathematical Sciences
University of Houston - Downtown
mailto: hodge...@uhd.edu


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Brian D. Ripley,  rip...@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel:  +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UKFax:  +44 1865 272595

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] with(x, Recall()) Crash

[Duncan Murdoch]

On 06/07/2010 5:38 PM, McGehee, Robert wrote:

R-devel,
I discovered a segfault in my R code that boiled down to my incorrect
use of the Recall() function embedded within a with() function. Since
segfaults are generally bad things, even when it's the user's fault for
writing nonsense code, I thought I'd pass along the offending code. I've
tested the crash on R 2.11.1 (on Linux and Mac), but not in devel
versions of R. 
  


Thanks for posting that.  It does still exist on R-devel, and as you 
say, it shouldn't.  I'll fix it.


Duncan Murdoch

HTH, Robert

  

x <- 1
with(x, Recall())


*** caught segfault ***
address 0x1db, cause 'memory not mapped'

Traceback:
 1: eval(expr, envir, enclos)
 2: Recall()
 3: eval(expr, envir, enclos)
 4: eval(substitute(expr), data, enclos = parent.frame())
 5: with.default(x, Recall())
 6: with(x, Recall())

  

R.version


   _
platform   x86_64-unknown-linux-gnu
arch   x86_64
os linux-gnu
system x86_64, linux-gnu
status
major  2
minor  11.1
year   2010
month  05
day31
svn rev52157
language   R
version.string R version 2.11.1 (2010-05-31)

Robert McGehee, CFA
Geode Capital Management, LLC
One Post Office Square, 28th Floor | Boston, MA | 02109
Tel: 617/392-8396Fax:617/476-6389
mailto:robert.mcge...@geodecapital.com


  

This e-mail, and any attachments hereto, are intended for use by the


addressee(s) only and may contain information that is (i) confidential
information of Geode Capital Management, LLC and/or its affiliates,
and/or (ii) proprietary information of Geode Capital Management, LLC
and/or its affiliates. If you are not the intended recipient of this
e-mail, or if you have otherwise received this e-mail in error, please
immediately notify me by telephone (you may call collect), or by e-mail,
and please permanently delete the original, any print outs and any
copies of the foregoing. Any dissemination, distribution or copying of
this e-mail is strictly prohibited. 


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] with(x, Recall()) Crash

[McGehee, Robert]

R-devel,
I discovered a segfault in my R code that boiled down to my incorrect
use of the Recall() function embedded within a with() function. Since
segfaults are generally bad things, even when it's the user's fault for
writing nonsense code, I thought I'd pass along the offending code. I've
tested the crash on R 2.11.1 (on Linux and Mac), but not in devel
versions of R. 

HTH, Robert

> x <- 1
> with(x, Recall())
*** caught segfault ***
address 0x1db, cause 'memory not mapped'

Traceback:
 1: eval(expr, envir, enclos)
 2: Recall()
 3: eval(expr, envir, enclos)
 4: eval(substitute(expr), data, enclos = parent.frame())
 5: with.default(x, Recall())
 6: with(x, Recall())

> R.version
   _
platform   x86_64-unknown-linux-gnu
arch   x86_64
os linux-gnu
system x86_64, linux-gnu
status
major  2
minor  11.1
year   2010
month  05
day31
svn rev52157
language   R
version.string R version 2.11.1 (2010-05-31)

Robert McGehee, CFA
Geode Capital Management, LLC
One Post Office Square, 28th Floor | Boston, MA | 02109
Tel: 617/392-8396Fax:617/476-6389
mailto:robert.mcge...@geodecapital.com


>This e-mail, and any attachments hereto, are intended for use by the
addressee(s) only and may contain information that is (i) confidential
information of Geode Capital Management, LLC and/or its affiliates,
and/or (ii) proprietary information of Geode Capital Management, LLC
and/or its affiliates. If you are not the intended recipient of this
e-mail, or if you have otherwise received this e-mail in error, please
immediately notify me by telephone (you may call collect), or by e-mail,
and please permanently delete the original, any print outs and any
copies of the foregoing. Any dissemination, distribution or copying of
this e-mail is strictly prohibited. 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] using svd in regression with arima

[Hodgess, Erin]

Dear R Developers:

Why is it that the singular value decomposition is used when running regression 
with arima, please?  I've been looking for a reference for that but have come 
up empty so far.

Thank you for any help.

Sincerely,
Erin


Erin M. Hodgess, PhD
Associate Professor
Department of Computer and Mathematical Sciences
University of Houston - Downtown
mailto: hodge...@uhd.edu


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] arr.ind argument to which.min and which.max

[Vincent Goulet]

Le lun. 5 juil. à 05:56, Martin Maechler a écrit :

>> "PatB" == Patrick Burns 
>>on Sun, 04 Jul 2010 09:43:44 +0100 writes:
> 
>PatB> Is there a reason that 'which.min' and
>PatB> 'which.max' don't have an 'arr.ind'
>PatB> argument?
> 
> well,  help(which.min)  tells you that they really were aimed at
> doing their job *fast* for vectors.
> 
> Of course you are right and a generalization to arrays might be
> convenient at times.
> 
>PatB> The context in which I wanted that was
>PatB> a grid search optimization, which seems
>PatB> like it would be reasonably common to me.
> 
> well, as the author of these two functions, I can only say
> 
>  "patches are welcome!"
> 
> and I think should be pretty simple, right ?
> You just have to do very simple remapping of the 1d index 'i' back
> to the array index, i.e., the same operation
> you need to transform seconds into days:hours:minutes:seconds
> {{and yes, we old-timers may recall that APL had an operator (I
>  think "T-bar") to do that ...}

Ah, a reference to APL! That made my day. :-)

APL was very big among actuaries for a long time. I heard recently that it was 
still in use on an old system in some small insurance company. I still have on 
my shelves the reference books for APL*Plus III for Windows (that I ran in 
OS/2) from Magnugistics; they have an actuarial formula on the cover.

I teach my students arbitrary base conversion to do just what the encode/decode 
operators (to call them by their names) were doing. I see this new arrayInd() 
sort of does the same thing now. Darn, this steals me an exercise!

Cheers,

Vincent


> 
> Martin Maechler, ETH Zurich
> 
> 
>PatB> -- 
>PatB> Patrick Burns
>PatB> pbu...@pburns.seanet.com
>PatB> http://www.burns-stat.com
>PatB> (home of 'Some hints for the R beginner'
>PatB> and 'The R Inferno')
> 
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] accessing underlying code

[Simon Urbanek]

On Jul 6, 2010, at 11:04 AM, Hodgess, Erin wrote:

> Dear R Developers:
> 
> Is there a way to look at the underlying code from such items as 
> R_setup_starma, please?
> 

Yes, of course, they are in the R sources (src/library/stats/src/pacf.c) - 
that's the beauty of open source.

Cheers,
Simon

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] accessing underlying code

[Matt Shotwell]

setup_starma is in src/library/stats/src/pacf.c

-Matt

On Tue, 2010-07-06 at 11:04 -0400, Hodgess, Erin wrote:
> Dear R Developers:
> 
> Is there a way to look at the underlying code from such items as 
> R_setup_starma, please?
> 
> Thanks,
> Erin
> 
> 
> Erin M. Hodgess, PhD
> Associate Professor
> Department of Computer and Mathematical Sciences
> University of Houston - Downtown
> mailto: hodge...@uhd.edu
> 
> 
>   [[alternative HTML version deleted]]
> 
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
-- 
Matthew S. Shotwell
Graduate Student
Division of Biostatistics and Epidemiology
Medical University of South Carolina
http://biostatmatt.com

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] accessing underlying code

[Hodgess, Erin]

Dear R Developers:

Is there a way to look at the underlying code from such items as 
R_setup_starma, please?

Thanks,
Erin


Erin M. Hodgess, PhD
Associate Professor
Department of Computer and Mathematical Sciences
University of Houston - Downtown
mailto: hodge...@uhd.edu


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Windows 64bit package build requires underscore hack?

[Jay Emerson]

I had a problem like this for a while with bigmemory et. al...  the check
went through to the very end, including successfully building the man pages,
then a number of shells popped up and the process died as described here.
The build was successful.  We never did figure out exactly why, but are
fairly sure it was related to our older, overly complicated in our
configuration process.  This included (among other things which I doubt are
related) invoking R from configure.win for the purpose of discovering
particulars of the platform/environment.

Once we streamlined the configuration process (and avoided calling R from
configure.win in particular), the problem vanished.

We never had this problem with the use of _ as you report.  I'm sure we have
many such characters, though perhaps not in exactly the same context.

Jay



To be more precise, while the build process worked, the check did not. When
it got to the point "* checking for portable compilation flags in Makevars
..."
a number of shells popped up (to run c:\Rtools\sh.exe) and the process died.

-- 
John W. Emerson (Jay)
Associate Professor of Statistics
Department of Statistics
Yale University
http://www.stat.yale.edu/~jay 

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] arr.ind argument to which.min and which.max

[Henrik Bengtsson]

On Tue, Jul 6, 2010 at 11:53 AM, Martin Maechler
 wrote:
>
> > "HenrikB" == Henrik Bengtsson 
> >     on Mon, 5 Jul 2010 22:53:59 +0200 writes:
>
>    HenrikB> ...and, of course, just after sending it off I found out that 
> from R
>    HenrikB> v2.11.0 there is now an arrayInd() in the 'base' package doing 
> exactly
>    HenrikB> the same thing.  See help(arrayInd).
>
> yes... and if you wondered *who* re-factored which() into its
> internal the arrayInd() part .
> .
> had not known about the R.utils function.
>
>
> []
>
>    >> On Mon, Jul 5, 2010 at 8:27 PM, Patrick Burns 
>  wrote:
>    >>> On 05/07/2010 10:56, Martin Maechler wrote:
>    >
>    > "PatB" == Patrick Burns
>    >     on Sun, 04 Jul 2010 09:43:44 +0100 writes:
>    
>         PatB>  Is there a reason that 'which.min' and
>         PatB>  'which.max' don't have an 'arr.ind'
>         PatB>  argument?
>    
>     well,  help(which.min)  tells you that they really were aimed at
>     doing their job *fast* for vectors.
>    
>     Of course you are right and a generalization to arrays might be
>     convenient at times.
>    
>         PatB>  The context in which I wanted that was
>         PatB>  a grid search optimization, which seems
>         PatB>  like it would be reasonably common to me.
>    
>     well, as the author of these two functions, I can only say
>    
>           "patches are welcome!"
>    
>     and I think should be pretty simple, right ?
>     You just have to do very simple remapping of the 1d index 'i' back
>     to the array index, i.e., the same operation
>     you need to transform seconds into days:hours:minutes:seconds
>     {{and yes, we old-timers may recall that APL had an operator (I
>       think "T-bar") to do that ...}
>    >>>
>    >>> I think the exercise is just to copy the definition of
>    >>> 'which' and add four characters.
>
> Well, yes.  But then, one reason for refactoring 'which' into its
> vector and arrayInd() part was that people could use arrayInd()
> on its own.
>
> Wouldn't it make more sense to call
>
>   arrayInd(which.min(mat), dim(mat))
>
> instead of
>            which.min(mat, arr.ind = TRUE)
>
> in the spirit of modularity, maintainability, ... ?
> Honestly, in my first reply I had forgotten about my own
> arrayInd() modularization
>
>    >>> If the order of the if condition were reversed, then
>    >>> possibly the slight reduction in speed of 'which.min'
>    >>> and 'which.max' would be more than made up for in the
>    >>> slight increase in speed of 'which'.
>
> thanks for the hint, but
>
>      "increase in speed of 'which'"  -- really, can you measure that?
>
> (I'll reverse the order anyway)
>
> If we are interested in speed increase, we should add an option
> to *not* work with dimnames at all (*) and if we have programmer
> time left, we could take it .Internal() and get a real
> boost... not now though.
>
> (*) I'm doing that for now, *and* I would like to change the
>    default behavior or arrayInd(), but of course *not* the
>    default behavior of which(),
>    to *not* attach dimnames to the result, by default.
>
>  I.e., I'm proposing to add   'useNames = FALSE' as argument to
>  arrayInd() but have  which() call arrayInd(..., useNames=TRUE).

I second this.

/Henrik

>
>  This is a back-compatibility change in arrayInd() -- which has
>  existed only since 2.11.0 anyway, so would seem ok, to me.
>
>  Opinions ?
>
> --
> Martin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] arr.ind argument to which.min and which.max

[Patrick Burns]

On 06/07/2010 10:53, Martin Maechler wrote:

[ ... ]



Wouldn't it make more sense to call

arrayInd(which.min(mat), dim(mat))

instead of
which.min(mat, arr.ind = TRUE)

in the spirit of modularity, maintainability, ... ?
Honestly, in my first reply I had forgotten about my own
arrayInd() modularization


Yes.  Then I guess the suggested change is to
put 'arrayInd' in the See Also and Examples for
'which.min' for dunderheads like me that don't
think of it themselves.



 >>>  If the order of the if condition were reversed, then
 >>>  possibly the slight reduction in speed of 'which.min'
 >>>  and 'which.max' would be more than made up for in the
 >>>  slight increase in speed of 'which'.

thanks for the hint, but

   "increase in speed of 'which'"  -- really, can you measure that?


I doubt it.



(I'll reverse the order anyway)

If we are interested in speed increase, we should add an option
to *not* work with dimnames at all (*) and if we have programmer
time left, we could take it .Internal() and get a real
boost... not now though.

(*) I'm doing that for now, *and* I would like to change the
 default behavior or arrayInd(), but of course *not* the
 default behavior of which(),
 to *not* attach dimnames to the result, by default.

   I.e., I'm proposing to add   'useNames = FALSE' as argument to
   arrayInd() but have  which() call arrayInd(..., useNames=TRUE).

   This is a back-compatibility change in arrayInd() -- which has
   existed only since 2.11.0 anyway, so would seem ok, to me.

   Opinions ?


I find it hard to believe that would cause
too much trauma.

Pat



--
Martin



--
Patrick Burns
pbu...@pburns.seanet.com
http://www.burns-stat.com
(home of 'Some hints for the R beginner'
and 'The R Inferno')

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] arr.ind argument to which.min and which.max

[Martin Maechler]

> "HenrikB" == Henrik Bengtsson 
> on Mon, 5 Jul 2010 22:53:59 +0200 writes:

HenrikB> ...and, of course, just after sending it off I found out that from 
R
HenrikB> v2.11.0 there is now an arrayInd() in the 'base' package doing 
exactly
HenrikB> the same thing.  See help(arrayInd).

yes... and if you wondered *who* re-factored which() into its
internal the arrayInd() part .
.
had not known about the R.utils function.


[]

>> On Mon, Jul 5, 2010 at 8:27 PM, Patrick Burns  
wrote:
>>> On 05/07/2010 10:56, Martin Maechler wrote:
> 
> "PatB" == Patrick Burns
>     on Sun, 04 Jul 2010 09:43:44 +0100 writes:
 
     PatB>  Is there a reason that 'which.min' and
     PatB>  'which.max' don't have an 'arr.ind'
     PatB>  argument?
 
 well,  help(which.min)  tells you that they really were aimed at
 doing their job *fast* for vectors.
 
 Of course you are right and a generalization to arrays might be
 convenient at times.
 
     PatB>  The context in which I wanted that was
     PatB>  a grid search optimization, which seems
     PatB>  like it would be reasonably common to me.
 
 well, as the author of these two functions, I can only say
 
       "patches are welcome!"
 
 and I think should be pretty simple, right ?
 You just have to do very simple remapping of the 1d index 'i' back
 to the array index, i.e., the same operation
 you need to transform seconds into days:hours:minutes:seconds
 {{and yes, we old-timers may recall that APL had an operator (I
   think "T-bar") to do that ...}
>>> 
>>> I think the exercise is just to copy the definition of
>>> 'which' and add four characters.

Well, yes.  But then, one reason for refactoring 'which' into its
vector and arrayInd() part was that people could use arrayInd()
on its own.

Wouldn't it make more sense to call

   arrayInd(which.min(mat), dim(mat))

instead of  
which.min(mat, arr.ind = TRUE)

in the spirit of modularity, maintainability, ... ?
Honestly, in my first reply I had forgotten about my own
arrayInd() modularization 

>>> If the order of the if condition were reversed, then
>>> possibly the slight reduction in speed of 'which.min'
>>> and 'which.max' would be more than made up for in the
>>> slight increase in speed of 'which'.

thanks for the hint, but

  "increase in speed of 'which'"  -- really, can you measure that?

(I'll reverse the order anyway)

If we are interested in speed increase, we should add an option
to *not* work with dimnames at all (*) and if we have programmer
time left, we could take it .Internal() and get a real
boost... not now though. 

(*) I'm doing that for now, *and* I would like to change the
default behavior or arrayInd(), but of course *not* the
default behavior of which(),
to *not* attach dimnames to the result, by default.

  I.e., I'm proposing to add   'useNames = FALSE' as argument to
  arrayInd() but have  which() call arrayInd(..., useNames=TRUE).

  This is a back-compatibility change in arrayInd() -- which has
  existed only since 2.11.0 anyway, so would seem ok, to me.

  Opinions ?

--
Martin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel