Re: [Rd] Apply .Rbuildignore before copying files in R CMD build

[Kevin Ushey]

Hi Alexey,

It's kind of gross, but if you need a local solution, you could put
your own executable `cp` script on your PATH somewhere, and have that
do something "smart" when it's invoked by R. One way to detect that
would be to check if the `R_CMD` environment variable is set. For
example:

https://gist.github.com/kevinushey/2351194ba540627831fa2d58073c097a

For posterity, this was filed some time ago on R's bug tracker at:

https://bugs.r-project.org/show_bug.cgi?id=17549

Best,
Kevin


On Wed, Aug 28, 2024 at 3:00 PM Alexey Sergushichev  wrote:
>
> Hi,
>
> Is there any reason why .Rbuildignore is not used before copying package
> files in R CMD build?
>
> For some of the packages I develop I have rather large directories with
> miscellaneous files for testing and other purposes. They are in my
> .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build
> from trying to copy them on the build process. Having them copied either
> breaks the build completely because /tmp directory gets out of space, or
> just slows it down a lot. So I wonder if there is a specific reason for
> this behavior and whether it could be change or controlled by some
> parameter.
>
> There is some discussion in the context of pkgbuild package:
> https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish
> workaround for that, which also does not work on Windows.
>
> Best,
> Alexey
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] API for converting LANGSXP to LISTSXP?

[Kevin Ushey]

In this case, Rcpp was internally converting (already-existing)
LISTSXPs to LANGSXPs using SET_TYPEOF in some places; the goal was to
allow Rcpp to continue doing this without using SET_TYPEOF just to
preserve existing behavior in an API-compliant way. I ended up doing
exactly what you suggested; thanks.

On Sat, Jul 6, 2024 at 8:09 PM Ivan Krylov  wrote:
>
> On Fri, 5 Jul 2024 15:27:50 +0800
> Kevin Ushey  wrote:
>
> > A common idiom in the R sources is to convert objects between LANGSXP
> > and LISTSXP by using SET_TYPEOF. However, this is soon going to be
> > disallowed in packages.
>
> Would you mind providing an example where a package needs to take an
> existing LISTSXP and convert it to a LANGSXP (or vice versa)? I think
> that Luke Tierney intended to replace the uses of
> SET_TYPEOF(allocList(...), LANGSXP) with allocLang(...).
>
> At least it's easy to manually convert between the two by replacing the
> head of the list using LCONS(CAR(list), CDR(list)) or CONS(CAR(lang),
> CDR(lang)): in a call, the rest of the arguments are ordinary LISTSXPs.
>
> --
> Best regards,
> Ivan

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] API for converting LANGSXP to LISTSXP?

[Kevin Ushey]

Hi,

A common idiom in the R sources is to convert objects between LANGSXP
and LISTSXP by using SET_TYPEOF. However, this is soon going to be
disallowed in packages. From what I can see, there isn't currently a
direct way to convert between these two object types using the
available API. At the R level, one can convert calls to pairlists
with:

> as.call(pairlist(as.symbol("rnorm"), 42))
rnorm(42)

However, the reverse is not possible:

> as.pairlist(call("rnorm", 42))
Error in as.pairlist(call("rnorm", 42)) :
  'language' object cannot be coerced to type 'pairlist'

One can do such a conversion via conversion to e.g. an intermediate R
list (VECSXP), but that seems wasteful. Would it make sense to permit
this coercion? Or, is there some other relevant API I'm missing?

For completeness, Rf_coerceVector() also emits the same error above
since it uses the same code path.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] changes in R-devel and zero-extent objects in Rcpp

[Kevin Ushey]

IMHO, this should be changed in both Rcpp and downstream packages:

1. Rcpp could check for out-of-bounds accesses in cases like these, and
emit an R warning / error when such an access is detected;

2. The downstream packages unintentionally making these out-of-bounds
accesses should be fixed to avoid doing that.

That is, I think this is ultimately a bug in the affected packages, but
Rcpp could do better in detecting and handling this for client packages
(avoiding a segfault).

Best,
Kevin


On Sat, Jun 8, 2024, 3:06 PM Ben Bolker  wrote:

>
> A change to R-devel (SVN r86629 or
>
> https://github.com/r-devel/r-svn/commit/92c1d5de23c93576f55062e26d446feface07250
> has changed the handling of pointers to zero-length objects, leading to
> ASAN issues with a number of Rcpp-based packages (the commit message
> reads, in part, "Also define STRICT_TYPECHECK when compiling inlined.c.")
>
>I'm interested in discussion from the community.
>
>Details/diagnosis for the issues in the lme4 package are here:
> https://github.com/lme4/lme4/issues/794, with a bit more discussion
> about how zero-length objects should be handled.
>
>The short(ish) version is that r86629 enables the
> CATCH_ZERO_LENGTH_ACCESS definition. This turns on the CHKZLN macro
> <
> https://github.com/r-devel/r-svn/blob/4ef83b9dc3c6874e774195d329cbb6c11a71c414/src/main/memory.c#L4090-L4104>,
>
> which returns a trivial pointer (rather than the data pointer that would
> be returned in the normal control flow) if an object has length 0:
>
> /* Attempts to read or write elements of a zero length vector will
> result in a segfault, rather than read and write random memory.
> Returning NULL would be more natural, but Matrix seems to assume
> that even zero-length vectors have non-NULL data pointers, so
> return (void *) 1 instead. Zero-length CHARSXP objects still have a
> trailing zero byte so they are not handled. */
>
>In the Rcpp context this leads to an inconsistency, where `REAL(x)`
> is a 'real' external pointer and `DATAPTR(x)` is 0x1, which in turn
> leads to ASAN warnings like
>
> runtime error: reference binding to misaligned address 0x0001
> for type 'const double', which requires 8 byte alignment
> 0x0001: note: pointer points here
>
> I'm in over my head and hoping for insight into whether this problem
> should be resolved by changing R, Rcpp, or downstream Rcpp packages ...
>
>cheers
> Ben Bolker
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Is ALTREP "non-API"?

[Kevin Ushey]

On Thu, Apr 25, 2024 at 4:24 AM Ivan Krylov via R-devel
 wrote:
>
> On Wed, 24 Apr 2024 15:31:39 -0500 (CDT)
> luke-tierney--- via R-devel  wrote:
>
> > We would be better off (in my view, not necessarily shared by others
> > in R-core) if we could get to a point where:
> >
> >  all entry points listed in installed header files can be used in
> >  packages, at least with some caveats;
> >
> >  the caveats are expressed in a standard way that is searchable,
> >  e.g. with a standardized comment syntax at the header file or
> >  individual declaration level.
>
> This sounds almost like Doxygen, although the exact syntax used to
> denote the entry points and the necessary comments is far from the most
> important detail at this point.

I'm guessing Doxygen would be overkill here? I think instead these
would just be structured comments that mark a particular function, or
set of functions, as part of the API -- and some automated tool could
then just pull those functions out into a list of API functions. Then,
we would have a single "source of truth" for what is in the API, and
could be seen at a glance by browsing / grepping the installed R
headers. I see this as a structured way of accomplishing what is
already being done to clarify whether functions are part of the API in
the R headers.

A similar approach would have macros like R_API, or with a bit more
specificity, maybe something like R_API(ALTREP), which would have no
actual definition -- they would exist in the source purely to mark
functions as part of (some subset of) the API. Or, similarly, anything
declared within a block like R_API {} would be considered part of the
API (to avoid the need to tag every declaration individually.) This
would at least make it easy to figure out what functions are part of
the R API, without requiring too much extra maintenance effort from
the R maintainers.

The other alternative I could imagine would be an installed header
like R_ext/API.h, which package authors who want to submit packages to
CRAN would be required to use, with direct usage of other headers
eventually being phased out. But that would be a larger maintenance
burden, unless its generation could be automated (e.g. from the
functions tagged above).

As a side note, it's worth stating that the set of API endpoints that
R Core wants to make available to CRAN packages, versus those that are
intended for other usages (e.g. applications embedding R), are
different sets. But I suspect this discussion is most relevant to R
package authors who wish to submit their packages to CRAN.

> > There are some 500 entry points in the R shared library that are in
> > the installed headers but not mentioned in WRE. These would need to
> > be reviewed and adjusted.
>
> Is there a way for outsiders to help? For example, would it help to
> produce the linking graph (package P links to entry points X, Y)? I
> understand that an entry point being unpopular doesn't mean it
> shouldn't be public (and the other way around), but combined with a
> list of entry points that are listed in WRE, such a graph could be
> useful to direct effort or estimate impact from interface changes.

I'm guessing the most welcome kinds of contributions would be
documentation? IMHO, "documenting an API" and "describing how an API
can be used" are somewhat separate endeavors. I believe R-exts does an
excellent job of the latter, but may not be the right vehicle for the
former. To that end, I believe it would be helpful to have some
structured API documentation as a separate R-api document. Each
documented function would have described inputs, outputs, whether
inputs + outputs require protection from the garbage collector, and
other important usage notes. This is something that I think could be
developed and maintained by community members, with members of the R
community submitting documentation for each of the available API
functions. Such an effort could be started independently from R Core,
but some guidance would be appreciated as far as (1) would such a
document eventually be accepted as part of the official R manuals, and
(2) if so, what would be required of such a document.

> --
> Best regards,
> Ivan
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [Feature Request] Hide API Key in download.file() / R's libcurl

[Kevin Ushey]

For cases like these, I think it would be more useful to have some
mechanism for associating URLs / hosts with credentials, and have R use
those credentials by default whenever accessing those URLs. Since
download.file() now supports custom headers, this could be a mechanism for
setting headers to be used by default when downloading files from
particular URLs. Then, users could do something like:

options(download.file.headers = list(
example.org = c(Authorization = "<...>")
))

And those headers would be used automatically by download.file() whenever
talking to that server.

All of that to say -- I think the better way forward would be to make it
easier to safely use authentication credentials in download.file(), rather
than just tooling in support for suppressing specific types of output.

Best,
Kevin


On Sat, Feb 3, 2024 at 1:33 PM Simon Urbanek 
wrote:
>
> Any reason why you didn't use quiet=TRUE to suppress that output?
>
> There is no official API structure for credentials in R repositories, so
R has no way of knowing which part of the URL are credentials as it is not
under R's purview - it could be part of the path or anything, so there is
no way R can reliably mask it. Hence it makes more sense for the user to
suppress the output if they think it may contain sensitive information -
and R supports that.
>
> If that's still not enough, then please make a concrete proposal that
defines exactly what kind processing you'd like to see under what
conditions - and how you think that will solve the problem.
>
> Cheers,
> Simon
>
>
>
> > On Feb 2, 2024, at 5:28 AM, Xinyi  wrote:
> >
> > Hi all,
> >
> > When trying to install a package from R using install.packages(), it
will
> > print out the full url address (of the remote repository) it was trying
to
> > access. A bit further digging shows it is from the in_do_curlDownload
> > method from R's libcurl
> > <
https://github.com/wch/r-source/blob/trunk/src/modules/internet/libcurl.c>:
> > install.packages() calls download.packages(), and download.packages()
calls
> > download.file(), which uses "libcurl" as its default method.
> >
> > This line from R mirror
> > <
https://github.com/wch/r-source/blob/trunk/src/modules/internet/libcurl.c#L772
>
> > ("if (!quiet) REprintf(_("trying URL '%s'\n"), url);")  prints the full
url
> > it is trying to access.
> >
> > This is totally fine for public urls without credentials, but in the
case
> > that a given url contains an API key, it poses security issues. For
> > example, if the getOption("repos") has been overridden to a
> > customized repository (protected by API keys), then
> >> install.packages("zoo")
> > Installing packages into '--removed local directory path--'
> > trying URL 'https://--removed userid--:--removed
> > api-ke...@repository-addresss.com:4443/.../src/contrib/zoo_1.8-12.tar.gz
'
> > Content type 'application/x-gzip' length 782344 bytes (764 KB)
> > ===
> > downloaded 764 KB
> >
> > * installing *source* package 'zoo' ...
> > -- further logs removed --
> >>
> >
> > I also tried several other options:
> >
> > 1. quite=1
> >> install.packages("zoo", quite=1)
> > It did hide the url, but it also hid all other useful information.
> > 2. method="curl"
> >> install.packages("zoo", method="curl")
> > This does not print the url when the download is successful, but if
there
> > were any errors, it still prints the url with API key in it.
> > 3. method="wget"
> >> install.packages("zoo", method="wget")
> > This hides API key by *password*, but I wasn't able to install packages
> > with this method even with public repos, with the error "Warning:
unable to
> > access index for repository https://cloud.r-project.org/src/contrib/4.3:
> > 'wget' call had nonzero exit status"
> >
> >
> > In other dynamic languages' package managers like Python's pip, API keys
> > are hidden by default since pip 18.x in 2018, and masked by "" from
pip
> > 19.x in 2019, see below examples. Can we get a similar default
behaviour in
> > R?
> >
> > 1. with pip 10.x
> > $ pip install numpy -v # API key was not hided
> > Looking in indexes:  https://--removed userid--:--removed
> > api-ke...@repository-addresss.com:4443/.../pypi/simple
> > 2. with pip 18.x # All credentials are removed by pip
> > $ pip install numpy -v
> > Looking in indexes:  https://repository-addresss.com:4443/
> > .../pypi/simple
> > 3. with pip 19.x onwards # userid is kept, API key is replaced by 
> > $ pip install numpy -v
> > Looking in indexes:  https://userid:@
> > repository-addresss.com:4443/.../pypi/simple
> >
> >
> > I was instructed by https://www.r-project.org/bugs.html that I should
get
> > some discussion on r-devel before filing a feature request. So looking
> > forward to comments/suggestions.
> >
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

[[alternative HTML version deleted]]

___

Re: [Rd] 答复: R-4.3 version list.files function could not work correctly in chinese

[Kevin Ushey]

Just to rule it out... is it possible that R is listing these files
successfully, but is not printing the Chinese characters in those
names for some reason?

Using your example, what is the output of:

f <- list.files(a, recursive = T)
nchar(f)

Does the reported number of characters match what you see?

Best,
Kevin

On Mon, Aug 14, 2023 at 12:32 AM 叶月光  wrote:
>
> Rterm.exe  test result。
>
> > a  =  readline()
> D:\Project_Delivery\
> >
> >
> >  list.files(a,recursive = T)
> [1] "2022(1).xlsx"
> [2] "conf_custom_wf_wt_map_202308091545.csv"
> [3] ".R"
> [4] ".xlsx"
> [5] "_.xlsx"
> [6] ".xlsx"
> [7] " (3).xlsx"
> [8] "20230222113605379(1).xlsx"
> [9] "_2022_20230811.docx"
>
> All  the file names which contains the Chinese  can not be printed.
> The result of RGUI and RStudio are the same:
>
>
> > a  =  readline()
> D:\Project_Delivery\广核
> >
> >
> >  list.files(a,recursive = T)
> [1] "2022年度金风慧能第三方模型预警明细表(1).xlsx" "conf_custom_wf_wt_map_202308091545.csv"
>
>
> 发件人: yu gong [mailto:yug...@outlook.com]
> 发送时间: 2023年8月13日 17:36
> 收件人: 叶月光 ; r-devel@r-project.org
> 主题: Re: R-4.3 version list.files function could not work correctly in chinese
>
> Could you test it on RGui and Rterm first, see it work or not. then try 
> RStudio?
>
> 
> From: R-devel 
> mailto:r-devel-boun...@r-project.org>> on 
> behalf of 叶月光 mailto:yeyuegu...@goldwind.com>>
> Sent: Friday, August 11, 2023 11:41
> To: r-devel@r-project.org 
> mailto:r-devel@r-project.org>>
> Subject: [Rd] R-4.3 version list.files function could not work correctly in 
> chinese
>
>  ��ã�
>   
> R-4.3�汾�°汾��R��list.files��ȡȫ���ļ���Ϣ��BUG�ж���û��ᵽ��⣬ϣ���Ľ���
> r4.3�汾��dir��ȡȫ���ļ� - COS��̳ | ͳ��֮�� | ͳ���ݿ�ѧ��̳ 
> (cosx.org)
>   ���ļ��б��ȡ��ʱ��
>
> [[alternative HTML version deleted]]
> 以下内容是邮件系统安全提示：
> 在电子邮件中索要个人信息、账号密码、银行卡信息、求助、补助、钱款转账等情况为“钓鱼邮件”或者“病毒邮件”，无需响应，并请立即删除。
> 如遇到邮件安全问题，请联系数字化中心 itsecur...@goldwind.com。
>
> ———
> Email system security tips：
> The use of emails to collect personal information, account passwords, bank 
> card information, help, subsidies, money transfers, etc. is "phishing email" 
> or "virus email", no response is required, and please delete it immediately.
> If you encounter email security issues, please contact 
> itsecur...@goldwind.com.
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Multiple Assignment built into the R Interpreter?

[Kevin Ushey]

FWIW, it's possible to get fairly close to your proposed semantics
using the existing metaprogramming facilities in R. I put together a
prototype package here to demonstrate:

https://github.com/kevinushey/dotty

The package exports an object called `.`, with a special `[<-.dot` S3
method which enables destructuring assignments. This means you can
write code like:

.[nr, nc] <- dim(mtcars)

and that will define 'nr' and 'nc' as you expect.

As for R CMD check warnings, you can suppress those through the use of
globalVariables(), and that can also be automated within the package.
The 'dotty' package includes a function 'dotify()' which automates
looking for such usages in your package, and calling globalVariables()
so that R CMD check doesn't warn. In theory, a similar technique would
be applicable to other packages defining similar operators (zeallot,
collapse).

Obviously, globalVariables() is a very heavy hammer to swing for this
issue, but you might consider the benefits worth the tradeoffs.

Best,
Kevin

On Sat, Mar 11, 2023 at 2:53 PM Duncan Murdoch  wrote:
>
> On 11/03/2023 4:42 p.m., Sebastian Martin Krantz wrote:
> > Thanks Duncan and Ivan for the careful thoughts. I'm not sure I can
> > follow all aspects you raised, but to give my limited take on a few:
> >
> >> your proposal violates a very basic property of the  language, i.e. that 
> >> all statements are expressions and have a value.  > What's the value of 1 
> >> + (A, C = init_matrices()).
> >
> > I'm not sure I see the point here. I evaluated 1 + (d = dim(mtcars); nr
> > = d[1]; nc = d[2]; rm(d)), which simply gives a syntax error,
>
>
>d = dim(mtcars); nr = d[1]; nc = d[2]; rm(d)
>
> is not a statement, it is a sequence of 4 statements.
>
> Duncan Murdoch
>
>   as the
> > above expression should. `%=%` assigns to
> > environments, so 1 + (c("A", "C") %=% init_matrices()) returns
> > numeric(0), with A and C having their values assigned.
> >
> >> suppose f() returns list(A = 1, B = 2) and I do  > B, A <- f() > Should 
> >> assignment be by position or by name?
> >
> > In other languages this is by position. The feature is not meant to
> > replace list2env(), and being able to rename objects in the assignment
> > is a vital feature of codes
> > using multi input and output functions e.g. in Matlab or Julia.
> >
> >> Honestly, given that this is simply syntactic sugar, I don't think I would 
> >> support it.
> >
> > You can call it that, but it would be used by almost every R user almost
> > every day. Simple things like nr, nc = dim(x); values, vectors =
> > eigen(x) etc. where the creation of intermediate objects
> > is cumbersome and redundant.
> >
> >> I see you've already mentioned it ("JavaScript-like"). I think it would  
> >> fulfil Sebastian's requirements too, as long as it is considered "true 
> >> assignment" by the rest of the language.
> >
> > I don't have strong opinions about how the issue is phrased or
> > implemented. Something like [t, n] = dim(x) might even be more clear.
> > It's important though that assignment remains by position,
> > so even if some output gets thrown away that should also be positional.
> >
> >>  A <- 0  > [A, B = A + 10] <- list(1, A = 2)
> >
> > I also fail to see the use of allowing this. something like this is an
> > error.
> >
> >> A = 2
> >> (B = A + 1) <- 1
> > Error in (B = A + 1) <- 1 : could not find function "(<-"
> >
> > Regarding the practical implementation, I think `collapse::%=%` is a
> > good starting point. It could be introduced in R as a separate function,
> > or `=` could be modified to accommodate its capability. It should be
> > clear that
> > with more than one LHS variables the assignment is an environment level
> > operation and the results can only be used in computations once assigned
> > to the environment, e.g. as in 1 + (c("A", "C") %=% init_matrices()),
> > A and C are not available for the addition in this statement. The
> > interpretor then needs to be modified to read something like nr, nc =
> > dim(x) or [nr, nc] = dim(x). as an environment-level multiple assignment
> > operation with no
> > immediate value. Appears very feasible to my limited understanding, but
> > I guess there are other things to consider still. Definitely appreciate
> > the responses so far though.
> >
> > Best regards,
> >
> > Sebastian
> >
> >
> >
> >
> >
> > On Sat, 11 Mar 2023 at 20:38, Duncan Murdoch  > > wrote:
> >
> > On 11/03/2023 11:57 a.m., Ivan Krylov wrote:
> >  > On Sat, 11 Mar 2023 11:11:06 -0500
> >  > Duncan Murdoch  > > wrote:
> >  >
> >  >> That's clear, but your proposal violates a very basic property
> > of the
> >  >> language, i.e. that all statements are expressions and have a value.
> >  >
> >  > How about reframing this feature request from multiple assignment
> >  > (which does go contrary to "everything has only one value, even
> > if it's
> > 

Re: [Rd] Unnecessary note when import only used in arg definition

[Kevin Ushey]

Hi Antoine,

Maybe I'm misunderstanding, but I think the warning is saying that
you've declared the package dependency in the DESCRIPTION file, but
you haven't actually imported the package (or any functions) in your
package NAMESPACE file?

I put together an example package that I think satisfies the point
you're describing, and I don't see any R CMD check warnings (using R
4.2.2).

https://github.com/kevinushey/imports.example

As I understand it, if you declare a package dependency in the
DESCRIPTION file, you need to clarify how you're using the package in
the NAMESPACE file -- e.g. what symbols you want to import, and so on.

Best,
Kevin

On Mon, Feb 6, 2023 at 6:43 AM Antoine Fabri  wrote:
>
> Dear r-devel,
>
> When a package is only used in an argument definition, e.g :
>
>
> f <- function(test = testthat::is_testing()) {
>
>   if (test) 1 else 2
>
> }
>
>
> R CMD CHECK gives us a note: "Namespace in Imports field not imported from:
> 'testthat'"
>
>
> This incites me to remove the package from the Imports field but that'll
> make my package brittle.
>
>
> I noted I'm not the first one having the issue (
> https://github.com/r-lib/devtools/issues/2456 ) and I've seen some
> workarounds too, in particular Hadley Wickham suggests in 'R packages' to
> use the following construct :
>
>
> ignore_unused_imports <- *function*() {
>
>   aaapkg::aaa_fun
>
> }
>
>
> That's far from obvious though, and not very satisfying.
>
> Are there any downside to removing this note in this scenario? it makes
> little sense to me and incites wrong behaviour AFAIU.
>
>
> Thanks,
>
>
> Antoine
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] String interpolation [Was: string concatenation operator (revisited)]

[Kevin Ushey]

For what it's worth, you can also get 90% of the way there with:

f <- glue::glue
f("if you squint, this is a Python f-string")

Having this in an add-on package also makes it much easier to change
in response to user feedback; R packages have more freedom to make
backwards-incompatible changes.

That said, if something like this were to happen in R, my vote would
be an implementation in the parser that transformed f"string" into
something like 'interpolate("string")', so that f"string" would just
become syntactic sugar for already-existing code (and so such code
could remain debuggable, easy to reason about, etc without any changes
to R internals)

Thanks,
Kevin

On Tue, Dec 7, 2021 at 2:06 PM Simon Urbanek
 wrote:
>
> I don't think a custom type alone would work, because users would expect to 
> use such string anywhere a regular string can be used, and that's where the 
> problems start - the evaluation would have to happen at a point where it is 
> not expected since we can assume today that CHAR() doesn't evaluate. If it's 
> just construct that needs some function call to turn it into a real string, 
> then that's (from user's perspective) no different than glue() so I don't 
> think the users would see the benefit (admittedly, you could do a lot more 
> with such internal type, but not sure if the complexity is worth it).
>
> Cheers,
> Simon
>
>
>
> > On Dec 8, 2021, at 12:56 AM, Taras Zakharko  wrote:
> >
> > I fully agree! General string interpolation opens a gaping security hole 
> > and is accompanied by all kinds of problems and decisions. What I envision 
> > instead is something like this:
> >
> >   f”hello {name}”
> >
> > Which gets parsed by R to this:
> >
> >   (STRINTERPSXP (CHARSXP (PROMISE nil)))
> >
> > Basically, a new type of R language construct that still can be processed 
> > by packages (for customized interpolation like in cli etc.), with a default 
> > eval which is basically paste0(). The benefit here would be that this is 
> > eagerly parsed and syntactically checked, and that the promise code could 
> > carry a srcref. And of course, that you could pass an interpolated string 
> > expression lazily between frames without losing the environment etc… For 
> > more advanced applications, a low level string interpolation expression 
> > constructor could be provided (that could either parse a general string — 
> > at the user’s risk, or build it directly from expressions).
> >
> > — Taras
> >
> >

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Inconsistent is.list results on 'by' objects

[Kevin Ushey]

You can see this a bit more clearly with e.g.

> storage.mode(byy)
[1] "list"
> storage.mode(byy.empty)
[1] "logical"

So even though both objects have S3 class "by", they have a different
underlying internal storage mode (as simplifying the result of 'by'
has given you a 0-length logical, instead of a 0-length list).

Best,
Kevin

On Tue, Nov 16, 2021 at 10:21 AM Bill Dunlap  wrote:
>
> Try adding simplify=FALSE to the call to by().
>
> -Bill
>
> On Tue, Nov 16, 2021 at 4:04 AM Ofek Shilon  wrote:
>
> > Take this toy code:
> > df   <- data.frame(a=seq(10), b=rep(1:2, 5))
> > df.empty <- subset(df, a>10)
> > byy   <- by(data=df,   INDICES=df$b, FUN=function(x) x[1,])
> > byy.empty <- by(data=df.empty, INDICES=df.empty$b, FUN=function(x) x[1,])
> >
> > class(byy)   # "by"
> > class(byy.empty) # "by"
> >
> > is.list(byy)# TRUE
> > is.list(byy.empty)  # FALSE!
> >
> >
> > This behavior already messed up stuff elsewhere:
> > https://github.com/Rdatatable/data.table/issues/5258
> >
> > I'd say any questions about the class of an object (whether 'class' or
> > indirectly by 'is.**')
> > should not have the answers depend on the specific contents of the object.
> >
> > Does this qualify as an R bug? Am I missing something?
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R-devel (r81196) hanging at dchisq(large) (PR#13309)

[Kevin Ushey]

Do you see this same hang in a build of R with debug symbols? Can you
try running R with GDB, or even WinDbg or another debugger, to see
what the call stack looks like when the hang occurs? Does the hang
depend on the number of threads used by OpenBLAS?

On the off chance it's relevant, I've seen hangs / crashes when using
a multithreaded OpenBLAS with R on some Linux systems before, but
never found the time to isolate a root cause.

Best,
Kevin

On Tue, Nov 16, 2021 at 5:12 AM Avraham Adler  wrote:
>
> On Tue, Nov 16, 2021 at 8:43 AM Martin Maechler
>  wrote:
> >
> > > Avraham Adler
> > > on Tue, 16 Nov 2021 02:35:56 + writes:
> >
> > > I am building r-devel on Windows 10 64bit using Jeroen's mingw system,
> > > and I am finding that my make check-devel hangs on the above issue.
> > > Everything is vanila except that I am using OpenBLAS 0.3.18. I have
> > > been using OpenBLAS for over a decade and have not had this issue
> > > before. Is there anything I can do to dig deeper into this issue from
> > > my end? Could there be anything that changed in R-devel that may have
> > > triggered this? The bugzilla report doesn't have any code attached to
> > > it.
> >
> > > Thank you,
> > > Avi
> >
> > Hmm.. it would've be nice to tell a bit more, instead of having all
> > your readers to search links, etc.
> >
> > In the bugzilla bug report PR#13309
> > https://bugs.r-project.org/show_bug.cgi?id=13309 ,the example was
> >
> >  dchisq(x=Inf, df=10, ncp=1)
> >
> > I had fixed the bug 13 years ago, in svn rev 47005
> > with regression test in /tests/d-p-q-r-tests.R :
> >
> >
> > ## Non-central Chi^2 density for large x
> > stopifnot(0 == dchisq(c(Inf, 1e80, 1e50, 1e40), df=10, ncp=1))
> > ## did hang in 2.8.0 and earlier (PR#13309).
> >
> >
> > and you are seeing your version of R hanging at exactly this
> > location?
> >
> >
> > I'd bet quite a bit that the underlying code in these
> > non-central chi square computations *never* calls BLAS and hence
> > I cannot imagine how openBLAS could play a role.
> >
> > However, there must be something peculiar in your compiler setup,
> > compilation options, 
> > as of course the above regression test has been run 100s of
> > 1000s of times also under Windows in the last 13 years ..
> >
> > Last but not least (but really only vaguely related):
> >There is still a FIXME in the source code (but not about
> > hanging, but rather of loosing some accuracy in border cases),
> > see e.g. https://svn.r-project.org/R/trunk/src/nmath/dnchisq.c
> > and for that reason I had written an R version of that C code
> > even back in 2008 which I've made available in  CRAN package
> > DPQ  a few years ago (together with many other D/P/Q
> > distribution computations/approximations).
> >  -> https://cran.r-project.org/package=DPQ
> >
> > Best,
> > Martin
> >
>
> Hello, Martin.
>
> Apologies, I thought the PR # was sufficient. Yes, I am seeing this at
> this exact location. This is what I saw in d-p-q-r-tst-2.Rout.fail and
> I then ran d-p-q-r-tst.R line-by-line and R hung precisely after
> `stopifnot(0 == dchisq(c(Inf, 1e80, 1e50, 1e40), df=10, ncp=1))`.
>
> Is it at all possible that this has to do with the recent change from
> bd0 to ebd0 (PR #15628) [1]?
>
> For completeness, I ran all the code _beneath_ the call, and while
> nothing else cause an infinite loop, I posted what I believe may be
> unexpected results below,
>
> Thank you,
>
> Avi
>
> [1]: https://bugs.r-project.org/show_bug.cgi?id=15628
>
> > ## FIXME ?!:  MxM/2 seems +- ok ??
> > (dLmM <- dnbinom(xL, mu = 1, size = MxM))  # all NaN but the last
> Warning in dnbinom(xL, mu = 1, size = MxM) : NaNs produced
>  [1] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
> NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN   0
> > (dLpI <- dnbinom(xL, prob=1/2, size = Inf))#  ditto
> Warning in dnbinom(xL, prob = 1/2, size = Inf) : NaNs produced
>  [1] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
> NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN   0
> > (dLpM <- dnbinom(xL, prob=1/2, size = MxM))#  ditto
> Warning in dnbinom(xL, prob = 1/2, size = MxM) : NaNs produced
>  [1] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
> NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN   0
>
> > d <- dnbinom(x,  mu = mu, size = Inf) # gave NaN (for 0 and L), now all 0
> Warning in dnbinom(x, mu = mu, size = Inf) : NaNs produced
> > p <- pnbinom(x,  mu = mu, size = Inf) # gave all NaN, now uses ppois(x, mu)
> Warning in pnbinom(x, mu = mu, size = Inf) : NaNs produced
>
> > pp <- (0:16)/16
> > q <- qnbinom(pp, mu = mu, size = Inf) # gave all NaN
> > set.seed(1); NI <- rnbinom(32, mu = mu, size = Inf)# gave all NaN
> > set.seed(1); N2 <- rnbinom(32, mu = mu, size = L  )
> > stopifnot(exprs = {
> + all.equal(d, c(0.006737947, 0.033689735, 0.0842243375,
> 0.140373896, 0,0,0,0), tol = 9e-9)# 7.6e-10
> + all.equal(p, c(0.006737947, 0.040427682,

[Rd] small typo in internet.c

[Kevin Ushey]

Hello,

I spotted a small typo recently in R-devel:

$ rg "htto://"
src/modules/internet/internet.c
95: warning(_("the 'wininet' method of url() is deprecated for
htto:// and https:// URLs"));

I suspect 'htto://' should read 'http://'.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Corrupt internal row names when creating a data.frame with `attributes<-`

[Kevin Ushey]

Strictly speaking, I don't think this is a "corrupt" representation,
given that any APIs used to access that internal representation will
call abs() on the row count encoded within. At least, as far as I can
tell, there aren't any adverse downstream effects from having the row
names attribute encoded with this particular internal representation.

On the other hand, the documentation in ?.row_names_info states, for
the 'type' argument:

integer. Currently type = 0 returns the internal "row.names" attribute
(possibly NULL), type = 2 the number of rows implied by the attribute,
and type = 1 the latter with a negative sign for ‘automatic’ row
names.

so one could argue that it's incorrect in light of that documentation
(the row names are "automatic", but the row count is not marked with a
negative sign). Or perhaps this is a different "type" of internal
automatic row name, since it was generated from an already-existing
integer sequence rather than "automatically" in a call to
data.frame().

Kevin

On Sun, Feb 14, 2021 at 6:51 AM Davis Vaughan  wrote:
>
> Hi all,
>
> I believe that the internal row names object created at this line in
> `row_names_gets()` should be using `-n`, not `n`.
> https://github.com/wch/r-source/blob/b30641d3f58703bbeafee101f983b6b263b7f27d/src/main/attrib.c#L71
>
> This can currently generate corrupt internal row names when using
> `attributes<-` or `structure()`, which calls `attributes<-`.
>
> # internal row names are typically `c(NA, -n)`
> df <- data.frame(x = 1:3)
> .row_names_info(df, type = 0L)
> #> [1] NA -3
>
> # using `attributes()` materializes their non-internal form
> attrs <- attributes(df)
> attrs
> #> $names
> #> [1] "x"
> #>
> #> $class
> #> [1] "data.frame"
> #>
> #> $row.names
> #> [1] 1 2 3
>
> # let's make a data frame from scratch with `attributes<-`
> data <- list(x = 1:3)
> attributes(data) <- attrs
>
> # oh no!
> .row_names_info(data, type = 0L)
> #> [1] NA  3
>
> # Note: Must have `nrow(df) > 2` to demonstrate this bug, as otherwise
> # internal row names are not attempted to be created in the C level
> # `row_names_gets()`
>
> Thanks,
> Davis
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] the pipe |> and line breaks in pipelines

[Kevin Ushey]

I agree with Duncan that the right solution is to wrap the pipe
expression with parentheses. Having the parser treat newlines
differently based on whether the session is interactive, or on what
type of operator happens to follow a newline, feels like a pretty big
can of worms.

I think this (or something similar) would accomplish what you want
while still retaining the nice aesthetics of the pipe expression, with
a minimal amount of syntax "noise":

result <- (
  data
|> op1()
|> op2()
)

For interactive sessions where you wanted to execute only parts of the
pipeline at a time, I could see that being accomplished by the editor
-- it could transform the expression so that it could be handled by R,
either by hoisting the pipe operator(s) up a line, or by wrapping the
to-be-executed expression in parentheses for you. If such a style of
coding became popular enough, I'm sure the developers of such editors
would be interested and willing to support this ...

Perhaps more importantly, it would be much easier to accomplish than a
change to the behavior of the R parser, and it would be work that
wouldn't have to be maintained by the R Core team.

Best,
Kevin

On Wed, Dec 9, 2020 at 11:34 AM Timothy Goodman  wrote:
>
> If I type my_data_frame_1 and press Enter (or Ctrl+Enter to execute the
> command in the Notebook environment I'm using) I certainly *would* expect R
> to treat it as a complete statement.
>
> But what I'm talking about is a different case, where I highlight a
> multi-line statement in my notebook:
>
> my_data_frame1
> |> filter(some_conditions_1)
>
> and then press Ctrl+Enter.  Or, I suppose the equivalent would be to run an
> R script containing those two lines of code, or to run a multi-line
> statement like that from the console (which in RStudio I can do by pressing
> Shift+Enter between the lines.)
>
> In those cases, R could either (1) Give an error message [the current
> behavior], or (2) understand that the first line is meant to be piped to
> the second.  The second option would be significantly more useful, and is
> almost certainly what the user intended.
>
> (For what it's worth, there are some languages, such as Javascript, that
> consider the first token of the next line when determining if the previous
> line was complete.  JavaScript's rules around this are overly complicated,
> but a rule like "a pipe following a line break is treated as continuing the
> previous line" would be much simpler.  And while it might be objectionable
> to treat the operator %>% different from other operators, the addition of
> |>, which isn't truly an operator at all, seems like the right time to
> consider it.)
>
> -Tim
>
> On Wed, Dec 9, 2020 at 3:12 AM Duncan Murdoch 
> wrote:
>
> > The requirement for operators at the end of the line comes from the
> > interactive nature of R.  If you type
> >
> >  my_data_frame_1
> >
> > how could R know that you are not done, and are planning to type the
> > rest of the expression
> >
> >%>% filter(some_conditions_1)
> >...
> >
> > before it should consider the expression complete?  The way languages
> > like C do this is by requiring a statement terminator at the end.  You
> > can also do it by wrapping the entire thing in parentheses ().
> >
> > However, be careful: Don't use braces:  they don't work.  And parens
> > have the side effect of removing invisibility from the result (which is
> > a design flaw or bonus, depending on your point of view).  So I actually
> > wouldn't advise this workaround.
> >
> > Duncan Murdoch
> >
> >
> > On 09/12/2020 12:45 a.m., Timothy Goodman wrote:
> > > Hi,
> > >
> > > I'm a data scientist who routinely uses R in my day-to-day work, for
> > tasks
> > > such as cleaning and transforming data, exploratory data analysis, etc.
> > > This includes frequent use of the pipe operator from the magrittr and
> > dplyr
> > > libraries, %>%.  So, I was pleased to hear about the recent work on a
> > > native pipe operator, |>.
> > >
> > > This seems like a good time to bring up the main pain point I encounter
> > > when using pipes in R, and some suggestions on what could be done about
> > > it.  The issue is that the pipe operator can't be placed at the start of
> > a
> > > line of code (except in parentheses).  That's no different than any
> > binary
> > > operator in R, but I find it's a source of difficulty for the pipe
> > because
> > > of how pipes are often used.
> > >
> > > [I'm assuming here that my usage is fairly typical of a lot of users; at
> > > any rate, I don't think I'm *too* unusual.]
> > >
> > > === Why this is a problem ===
> > >
> > > It's very common (for me, and I suspect for many users of dplyr) to write
> > > multi-step pipelines and put each step on its own line for readability.
> > > Something like this:
> > >
> > >### Example 1 ###
> > >my_data_frame_1 %>%
> > >  filter(some_conditions_1) %>%
> > >  inner_join(my_data_frame_2, by = some_columns_1) %>%
> > >

Re: [Rd] New pipe operator

[Kevin Ushey]

IMHO the use of anonymous functions is a very clean solution to the
placeholder problem, and the shorthand lambda syntax makes it much
more ergonomic to use. Pipe implementations that crawl the RHS for
usages of `.` are going to be more expensive than the alternatives. It
is nice that the `|>` operator is effectively the same as a regular R
function call, and given the identical semantics could then also be
reasoned about the same way regular R function calls are.

I also agree usages of the `.` placeholder can make the code more
challenging to read, since understanding the behavior of a piped
expression then requires scouring the RHS for usages of `.`, which can
be challenging in dense code. Piping to an anonymous function makes
the intent clear to the reader: the programmer is likely piping to an
anonymous function because they care where the argument is used in the
call, and so the reader of code should be aware of that.

Best,
Kevin



On Mon, Dec 7, 2020 at 10:35 AM Gabor Grothendieck
 wrote:
>
> On Mon, Dec 7, 2020 at 12:54 PM Duncan Murdoch  
> wrote:
> > An advantage of the current implementation is that it's simple and easy
> > to understand.  Once you make it a user-modifiable binary operator,
> > things will go kind of nuts.
> >
> > For example, I doubt if there are many users of magrittr's pipe who
> > really understand its subtleties, e.g. the example in Luke's paper where
> > 1 %>% c(., 2) gives c(1,2), but 1 %>% c(c(.), 2) gives c(1, 1, 2). (And
> > I could add 1 %>% c(c(.), 2, .) and  1 %>% c(c(.), 2, . + 2)  to
> > continue the fun.)
>
> The rule is not so complicated.  Automatic insertion is done unless
> you use dot in the top level function or if you surround it with
> {...}.  It really makes sense since if you use gsub(pattern,
> replacement, .) then surely you don't want automatic insertion and if
> you surround it with { ... } then you are explicitly telling it not
> to.
>
> Assuming the existence of placeholders a possible simplification would
> be to NOT do automatic insertion if { ... } is used and to use it
> otherwise although personally having used it for some time I find the
> existing rule in magrittr generally does what you want.
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] `basename` and `dirname` change the encoding to "UTF-8"

[Kevin Ushey]

Did you test with R 4.0.2 or R-devel? A bug related to this issue was
recently fixed:

https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17833

Best,
Kevin

On Mon, Jun 29, 2020 at 11:51 AM Duncan Murdoch
 wrote:
>
> On 29/06/2020 10:39 a.m., Johannes Rauh wrote:
> > Dear R Developers,
> >
> > I noticed that `basename` and `dirname` always return "UTF-8" on Windows 
> > (tested with R-4.0.0 and R-3.6.3):
> >
> >> p <- "Föö/Bär"
> >> Encoding(p)
> > [1] "latin1"
> >> Encoding(dirname(p))
> > [1] "UTF-8"
> >> Encoding(basename(p))
> > [1] "UTF-8"
> >
> > Is this on purpose?  At least I did not find any relevant comment in the 
> > documentation of `dirname`/`basename`.
> >
> > Background: I'm currently struggeling with a directory name containing a 
> > latin1-character.  (I know that this is a bad idea, but I did not create 
> > the directory and I cannot rename it.)  I now want to pass a 
> > latin1-directory name to a function, which internally uses 
> > `tools::makeLazyLoadDB`.  At that point, internally, `dirname` is called, 
> > which changes the encoding, and things break.  If I use `debug` to halt the 
> > processing and "fix" the encoding, things work as expected.
> >
> > So, if possible, I would prefer that `dirname` and `basename` preserve the 
> > encoding.
>
> Actually, makeLazyLoadDB isn't exported from tools, so strictly speaking
> you shouldn't be calling it.  Or perhaps you have a good reason to call
> it, and should be asking for it to be exported, or you are calling a
> published function which calls it:  in either case it should probably be
> fixed to accept UTF-8.
>
> But it doesn't call dirname or basename, so maybe the function that
> calls it is the one that needs fixing.
>
> In any case, while asking dirname() and basename() to preserve the
> encoding sounds reasonable, it seems like it would just be covering up a
> deeper problem.
>
> Duncan Murdoch
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Possible Bug: file.exists() Function. Due to UTF-8 Encoding differences on Windows between R 4.0.1 and R 3.6.3?

[Kevin Ushey]

Hi Juan,

For bug reports to R, you should attempt to create a
minimally-reproducible example, using only R's builtin facilities and
not any other addon packages. Given your report, it's not clear
whether the issue lies within renv or truly is caused by a change in R
4.0.0.

Also note that you have not supplied a minimally reproducible example.
If at all possible, you should be able to supply some code that
reproduces the issue -- ideally, one should be able to just copy +
paste the code into an R session to see the issue arise. Presumably,
if the issue is indeed in base R, then you should be able to supply a
reproducible example of the form:

path <- "path/that/causes/issue"
file.exists(path)

Alternatively, if you can distill this into a minimally-reproducible
example that does require renv, then you should report that to the
maintainer of renv (me), not this mailing list.

Best,
Kevin

On Wed, Jun 10, 2020 at 4:55 AM Dirk Eddelbuettel  wrote:
>
>
> On 10 June 2020 at 13:06, Juan Telleria Ruiz de Aguirre wrote:
> | I am having an issue with the renv package and R 4.0.1, which I
> | suspect is related to base R and not the renv package itself, as with
> | R 3.6.3 such an "error" does not appear.
>
> So a bug in `renv` as it does not account for changes in R 4.0.0 ?
>
> Stuff happens. I just fixed an 'change in R 4.0.0' for one small aspect of
> Rcpp(Armadillo) (namely the change in package.skeleton() and NAMESPACE).
>
> Dirk
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rtools and R 4.0.0?

[Kevin Ushey]

That's great to see, although I suspect it's still a speculative
change and could be backed out if any non-trivial issues were
encountered.

Regardless, I would like to thank R core, CRAN, and Jeroen for all of
the time that has gone into creating and validating this new
toolchain. This is arduous work at an especially arduous time, so I'd
like to voice my appreciation for all the time and energy they have
spent on making this possible.

Best,
Kevin

On Tue, Apr 7, 2020 at 7:47 AM Dirk Eddelbuettel  wrote:
>
>
> There appears to have been some progress on this matter:
>
> -Note that @command{g++} 4.9.x (as used for @R{} on Windows up to 3.6.x)
> +Note that @command{g++} 4.9.x (as used on Windows prior to @R{} 4.0.0)
>
> See SVN commit r78169 titled 'anticipate change in Windows toolchain', or the
> mirrored git commit at
> https://github.com/wch/r-source/commit/bd674e2b76b2384169424e3d899fbfb5ac174978
>
> Dirk
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Rtools and R 4.0.0?

[Kevin Ushey]

Hello,

Has a decision been made yet as to whether R 4.0.0 on Windows is going
to be built using the new gcc8 toolchain (described at
https://cran.r-project.org/bin/windows/testing/rtools40.html)?

>From the sidelines, I can see that the toolchain is being used to
build and test packages on CRAN; if there are any remaining issues
that I can help to try and run down (either in R or any CRAN packages)
I'd be happy to try and help.

FWIW, I'm excited at the prospect at seeing a new toolchain for
Windows, since it would imply support for C++17 and so it would become
easier for CRAN packages to depend on the newer C++ standard.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: should base R have a piping operator ?

[Kevin Ushey]

IMHO, if base R were to include a pipe operator, I think it should be much
simpler than the magrittr pipe. It should satisfy the property that:

x |> f(...)   is equivalent to   f(x, ...)

Except, perhaps, in terms of when the promise for 'x' gets forced. We
shouldn't need to mess with bindings in environments to make that work.

My understanding is that the '.' placeholder is used so that the magrittr
pipe can be adapted to functions that aren't endomorphic or otherwise
easily pipeable. I would argue that:

1. Users could just create their own pipable wrapper functions if so
required, or
2. Users could use magrittr to get some of the 'extensions' to the pipe
operator (with the noted caveats).

Best,
Kevin

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] ?Syntax wrong about `?`'s precedence ?

[Kevin Ushey]

See also: https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16710

On Fri, Aug 30, 2019 at 9:02 AM William Dunlap via R-devel
 wrote:
>
> Precedence is a property of the parser and has nothing to do with the
> semantics assigned to various symbols.  Using just core R functions you can
> see the precedence of '?' is between those of '=' and '<-'.
>
> > # '=' has lower precedence than '?'
> > str(as.list(parse(text="a ? b = c")[[1]]))
> List of 3
>  $ : symbol =
>  $ : language `?`(a, b)
>  $ : symbol c
> > str(as.list(parse(text="a = b ? c")[[1]]))
> List of 3
>  $ : symbol =
>  $ : symbol a
>  $ : language `?`(b, c)
> > # '<-' has higher precedence than '?'
> > str(as.list(parse(text="a ? b <- c")[[1]]))
> List of 3
>  $ : symbol ?
>  $ : symbol a
>  $ : language b <- c
> > str(as.list(parse(text="a <- b ? c")[[1]]))
> List of 3
>  $ : symbol ?
>  $ : language a <- b
>  $ : symbol c
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
>
> On Fri, Aug 30, 2019 at 4:41 AM Stephen Ellison 
> wrote:
>
> > > From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Ant F
> > > Sent: 29 August 2019 12:06
> > > To: r-devel@r-project.org
> > > Subject: [Rd] ?Syntax wrong about `?`'s precedence ?
> > > ...
> > > See the following example :
> > >
> > > `?` <- `+`
> >
> > I'm curious; What did you expect to happen if you replace the function '?'
> > with the operator '+' ?
> > ? is surely now being evaluated as a user-defined function and not as an
> > operator.
> > Would you expect the results of doing that to be the same as evaluation
> > without replacement?
> >
> > S Ellison
> >
> >
> >
> >
> > ***
> > This email and any attachments are confidential. Any u...{{dropped:10}}
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Possible bug in `class<-` when a class-specific '[[.' method is defined

[Kevin Ushey]

When RStudio builds the Environment pane, it will evaluate some R code
on objects in your global environment (as you have seen). In
particular, for better or worse, it calls `str()` on objects in the
global environment, to get a short text summary of the object.

So, to reproduce what you're seeing in a plain R session, you can check:

> counttt
[1] 0
> str(df)
Classes 'MYCLASS' and 'data.frame': 5 obs. of  4 variables:
 $ V1: int  1 2 3 4 5
 $ V2: int  6 7 8 9 10
 $ V3: int  11 12 13 14 15
 $ V4: int  16 17 18 19 20
> counttt
[1] 9

Newer versions will allow you to disable the Environment pane if you
so desire, since calling `str()` can have these kinds of undesirable
side effects in some cases.

In general though, if you're reporting a bug in R (as opposed to
RStudio) it's best to verify that you can reproduce the issue in a
'plain' R session (e.g. in the terminal) to be absolutely sure you're
seeing an R issue as opposed to an RStudio issue, as I imagine the
last thing R Core wants to do is spend time tracking down an issue
only to find it was in RStudio rather than R itself...

If you want to follow up further with the RStudio team I'd recommend
making a post at https://community.rstudio.com/c/rstudio-ide.

Thanks,
Kevin

On Mon, Jul 15, 2019 at 7:59 AM Rui Barradas  wrote:
>
> Hello,
>
> Inline.
>
> Às 14:26 de 15/07/19, Duncan Murdoch escreveu:
> > On 15/07/2019 8:57 a.m., Rui Barradas wrote:
> >> Hello,
> >>
> >> Clean R 3.6.1 session on Ubuntu 19.04, RStudio 1.1.453. sessionInfo() at
> >> the end.
> >
> > That's not what I'd call a "clean session" with all those packages loaded:
>
> You are right, but when I wrote that it *was* clean. Then, for some
> reason I don't understand, RStudio loaded them all. Guess I'll have to
> check what is going on here.
>
> >
> >> loaded via a namespace (and not attached):
> >> [1] sos_2.0-0   nlme_3.1-140matrixStats_0.54.0
> >> [4] fs_1.2.7xts_0.11-2  usethis_1.5.0
> >> [7] lubridate_1.7.4 devtools_2.0.2  RColorBrewer_1.1-2
> >>[10] rprojroot_1.3-2 rbenchmark_1.0.0tools_3.6.1
> >>[13] backports_1.1.4 R6_2.4.0rpart_4.1-15
> >>[16] Hmisc_4.2-0 lazyeval_0.2.2  colorspace_1.4-1
> >>[19] nnet_7.3-12 npsurv_0.4-0withr_2.1.2
> >>[22] tidyselect_0.2.5gridExtra_2.3   prettyunits_1.0.2
> >>[25] processx_3.3.0  curl_3.3compiler_3.6.1
> >>[28] cli_1.1.0   htmlTable_1.13.1randomNames_1.4-0.0
> >>[31] dvmisc_1.1.3desc_1.2.0  tseries_0.10-46
> >>[34] scales_1.0.0checkmate_1.9.1 lmtest_0.9-36
> >>[37] fracdiff_1.4-2  mvtnorm_1.0-10  quadprog_1.5-6
> >>[40] callr_3.2.0 stringr_1.4.0   digest_0.6.18
> >>[43] foreign_0.8-71  rio_0.5.16  base64enc_0.1-3
> >>[46] stocks_1.1.4pkgconfig_2.0.2 htmltools_0.3.6
> >>[49] sessioninfo_1.1.1   readxl_1.3.1htmlwidgets_1.3
> >>[52] rlang_0.3.4 TTR_0.23-4  rstudioapi_0.10
> >>[55] quantmod_0.4-14 MLmetrics_1.1.1 zoo_1.8-5
> >>[58] zip_2.0.1   acepack_1.4.1   dplyr_0.8.0.1
> >>[61] car_3.0-2   magrittr_1.5Formula_1.2-3
> >>[64] Matrix_1.2-17   Rcpp_1.0.1  munsell_0.5.0
> >>[67] abind_1.4-5 stringi_1.4.3   forecast_8.6
> >>[70] yaml_2.2.0  carData_3.0-2   MASS_7.3-51.3
> >>[73] pkgbuild_1.0.3  plyr_1.8.4  grid_3.6.1
> >>[76] parallel_3.6.1  forcats_0.4.0   crayon_1.3.4
> >>[79] lattice_0.20-38 haven_2.1.0 splines_3.6.1
> >>[82] hms_0.4.2   knitr_1.22  ps_1.3.0
> >>[85] pillar_1.4.0pkgload_1.0.2   urca_1.3-0
> >>[88] glue_1.3.1  lsei_1.2-0  babynames_1.0.0
> >>[91] latticeExtra_0.6-28 data.table_1.12.2   remotes_2.0.4
> >>[94] cellranger_1.1.0testthat_2.1.0  gtable_0.3.0
> >>[97] purrr_0.3.2 assertthat_0.2.1ggplot2_3.1.1
> >> [100] openxlsx_4.1.0  xfun_0.6survey_3.35-1
> >> [103] survival_2.44-1.1   timeDate_3043.102   tibble_2.1.1
> >> [106] memoise_1.1.0   cluster_2.0.8   toOrdinal_1.1-0.0
> >> [109] fitdistrplus_1.0-14 brew_1.0-6
> >>
> >>
> >
> > However, even when I load almost all of those, I don't see the problem.
> > I've got the same R version, and a newer Rstudio version (mine is
> > 1.2.1335 on a Mac).  I couldn't load ] "latticeExtradata.table" and
> > "fitdistrplusbrew", and I didn't check my package versions against yours.
> >
> > I'd suspect the issue is with RStudio somehow, because it needs to do a
> > lot to maintain its environment view.  Do you see this when running R
> > from the command line?
> >
> > Duncan Murdoch
> >
>
> It's a RStudio thing.
> Tested with
>
> Rscript --vanilla test.R
>
>
> the result is the expected one.
> (test.R has the obvious contents.)
>
>
> Rui Barradas
>
> _

Re: [Rd] Fast way to call an R function from C++?

[Kevin Ushey]

Hi Jiefei,

Calling into R from C++ code is more complicated than one might think.
Please see Tomas Kalibera's post here:
https://developer.r-project.org/Blog/public/2019/03/28/use-of-c---in-packages/index.html

The Rcpp Function class is more expensive than a regular Rf_eval()
because it tries to prevent errors (longjmps) from jumping to the top
level, so that the C++ stack can be properly unwound. In addition,
your example also pays a cost for the indirect function calls used
with Rcpp attributes, since the generated R wrapper functions will
then use `.Call()` under the hood. You could also try to directly use
.Call() to avoid that extra R function call overhead.

Even then, R's dispatch system for primitive functions (like `^`) is
likely still going to be faster than what you get from the .Call()
interface, especially since some extra validation does occur when
using .Call(). In practice, that difference is usually negligible
outside of synthetic benchmarks.

Best,
Kevin

On Tue, Jun 18, 2019 at 10:41 AM King Jiefei  wrote:
>
> Hi,
>
> I'm looking for a most efficient way to call an R function from C++ in a
> package. I know there are two functions (`R_forceAndCall` and `Rf_eval`)
> that can do the "call" part, but both are slow compared to calling the same
> function in R. I also try to use Rcpp and it is the worse one. Here is my
> test code:
>
> C++ code:
> ```
> // [[Rcpp::export]]
> SEXP C_test1(SEXP f, SEXP x) {
> SEXP call =PROTECT(Rf_lang2(f, x));
> SEXP val = R_forceAndCall(call, 1, R_GlobalEnv);
> UNPROTECT(1);
> return val;
> }
>
> // [[Rcpp::export]]
> SEXP C_test2(SEXP expr, SEXP env) {
> SEXP val = Rf_eval(expr, env);
> return val;
> }
>
> // [[Rcpp::export]]
> SEXP C_test3(SEXP f,SEXP x) {
> Function fun(f);
> return fun(x);
> }
> ```
>
> R code:
> ```
> testFunc<-function(x){
>   x=x^2
>   return(x)
> }
> evn=new.env()
> evn$x=x
> expr=quote(testFunc(evn$x))
>
> testFunc(evn$x)
> C_test1(testFunc, evn$x)
> C_test2(expr,evn)
> C_test3(testFunc,evn$x)
> ```
>
> For the results, I run each function 1,000,000 times:
>
>- testFunc : 0.47 sec
>- C_test1 : 2.46 sec
>- C_test2 : 2.74 sec
>- C_test3 : 18.86 sec
>
> It is clear to see that calling an R function in R is the fast one, it is
> about 5X faster than ` R_forceAndCall ` and ` Rf_eval`. the latter two
> functions have a similar performance and using Rcpp is the worst one. Is it
> expected? Why is calling an R function from C++ much slower than calling
> the function from R? Is there any faster way to do the function call in C++?
>
> Best,
> Jiefei
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Converting non-32-bit integers from python to R to use bit64: reticulate

[Kevin Ushey]

I think a more productive conversation could be: what additions to R
would allow for user-defined types / classes that behave just like the
built-in vector types? As a motivating example, one cannot currently
use the 64bit integer objects from bit64 to subset data frames:

   > library(bit64); mtcars[as.integer64(1:3), ]
[1] mpg  cyl  disp hp   drat wt   qsec vs   am   gear carb
   <0 rows> (or 0-length row.names)

I think ALTREP presents a possibility here, in that we could have a
64bit integer ALTREP object that behaves either like an INTSXP or
REALSXP as necessary. But I'm not sure how we would handle large 64bit
integer values which won't fit in either an INTSXP or REALSXP (in the
REALSXP case, precision could be lost for values > 2^53).

One possibility would be to allow ALTREP objects to have a chance at
managing dispatch in some methods, so that (for example) in e.g.
data[], the ALTREP object has the opportunity to choose how
the data object should be subsetted. Of course, this implies wiring
through yet another dispatch mechanism through a category of primitive
/ internal functions, which could be expensive in terms of
implementation / maintenance... and I'm not sure if this could play
well with the existing S3 / S4 dispatch mechanisms.

FWIW, I think most commonly 64bit integers arise as e.g. database keys
/ IDs, and are typically just used for subsetting / reordering of data
as opposed to math. In these cases, converting the 64bit integers to a
character vector is typically a viable workaround, although it's much
slower.

Still, at least to me, it seems like there is likely a path forward
with ALTREP for 64bit integer vectors that can behave (more or less)
just like builtin R vectors.

Best,
Kevin

On Tue, Jun 4, 2019 at 9:34 AM Martin Maechler
 wrote:
>
> > Juan Telleria Ruiz de Aguirre
> > on Mon, 3 Jun 2019 06:50:17 +0200 writes:
>
> > Thank you Martin for giving to know and developing 'Rmpfr' library for
> > unlimited size integers (GNU C GMP) and arbitrary precision floats (GNU 
> C
> > MPFR):
>
> > https://cran.r-project.org/package=Rmpfr
>
> > My question is: In the long term (For R3.7.0 or R3.8.0):
>
> > Does it have sense that CMP substitutes INTSXP, and MPFR substitutes
> > REALSXP code? With this we would achieve that an integer is always an
> > integer, and a numeric double precision float always a numeric double
> > precision float, without sometimes casting underneath.
>
> > And would the R Community / R Ordinary Members would be willing to help 
> R
> > Core on such implementation (If has sense, and wants to be adopted)?
>
> No, such a change has "no sense" and hence won't be adopted (in
> this form):
>
> - INTSXP and REALSXP are part of the C API of R, and are well defined.
>   Changing them will almost surely break 100s and by
>   dependencies, probably 1000s of existing R packages.
>
> - I'm sure Python and other system do have fixed size "double
>   precision" vectors, because that's how you interface with all
>   pre-existing computational libraries,
>   and I am almost sure that support of arbitrary long integer
>   (or double) is via another class/type.
>
> - I know that Julia has adopted these (GMP and MPFR I think)
>   types and nicely interfaces them on a relatively "base" level.
>   With their nice class hierarchy (and very nice "S4 like" multi-argument
>   method dispatch for *all* functions) it can look quite
>   seemless for the user to work with these extended classes, but
>   they are not all identical to the basic "real"/"double" or "integer" 
> classes.
>
> - I'm not the expert here (but there are not so many experts
>   ..), but I'm pretty sure that adding new "basic types" in the
>   underlying C level seems not at all easy for R.  It would mean a big
>   break in all back compatibility -- which is conceivable --
>   and *may* also need a big rewrite of much of the R code base
>   which seems less conceivable in the mid term (2-3 years; long
>   term: > 5 years).
>
>
> > Thank you all! :)
>
> You are welcome.
>
> I think we should close this thread here,  unless some real
> experts join.
>
> Martin
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()

[Kevin Ushey]

I think it's worth saying that mclapply() works as documented: it
relies on forking, and so doesn't work well in environments where it's
unsafe to fork. This is spelled out explicitly in the documentation of
?mclapply:

It is strongly discouraged to use these functions in GUI or embedded
environments, because it leads to several processes sharing the same
GUI which will likely cause chaos (and possibly crashes). Child
processes should never use on-screen graphics devices.

I believe the expectation is that users who need more control over the
kind of cluster that's used for parallel computations would instead
create the cluster themselves with e.g. `makeCluster()` and then use
`clusterApply()` / `parLapply()` or other APIs as appropriate.

In environments where forking works, `mclapply()` is nice because you
don't need to think -- the process is forked, and anything available
in your main session is automatically available in the child
processes. This is a nice convenience for when you know it's safe to
fork R (and know what you're doing is safe to do within a forked
process). When it's not safe, it's better to prefer the other APIs
available for computation on a cluster.

Forking can be unsafe and dangerous, but it's also convenient and
sometimes that convenience can outweigh the other concerns.

Finally, I want to add: the onus should be on the front-end to work
well with R, and not the other way around. I don't think it's fair to
impose extra work / an extra maintenance burden on the R Core team for
something that's already clearly documented ...

Best,
Kevin


On Fri, Apr 12, 2019 at 6:04 PM Travers Ching  wrote:
>
> Hi Inaki,
>
> > "Performant"... in terms of what. If the cost of copying the data
> > predominates over the computation time, maybe you didn't need
> > parallelization in the first place.
>
> Performant in terms of speed.  There's no copying in that example
> using `mclapply` and so it is significantly faster than other
> alternatives.
>
> It is a very simple and contrived example, but there are lots of
> applications that depend on processing of large data and benefit from
> multithreading.  For example, if I read in large sequencing data with
> `Rsamtools` and want to check sequences for a set of motifs.
>
> > I don't see why mclapply could not be rewritten using PSOCK clusters.
>
> Because it would be much slower.
>
> > To implement copy-on-write, Linux overcommits virtual memory, and this
> >  is what causes scripts to break unexpectedly: everything works fine,
> > until you change a small unimportant bit and... boom, out of memory.
> > And in general, running forks in any GUI would cause things everywhere
> > to break.
>
> > I'm not sure how did you setup that, but it does complete. Or do you
> > mean that you ran out of memory? Then try replacing "x" with, e.g.,
> > "x+1" in your mclapply example and see what happens (hint: save your
> > work first).
>
> Yes, I meant that it ran out of memory on my desktop.  I understand
> the limits, and it is not perfect because of the GUI issue you
> mention, but I don't see a better alternative in terms of speed.
>
> Regards,
> Travers
>
>
>
>
> On Fri, Apr 12, 2019 at 3:45 PM Iñaki Ucar  wrote:
> >
> > On Fri, 12 Apr 2019 at 21:32, Travers Ching  wrote:
> > >
> > > Just throwing my two cents in:
> > >
> > > I think removing/deprecating fork would be a bad idea for two reasons:
> > >
> > > 1) There are no performant alternatives
> >
> > "Performant"... in terms of what. If the cost of copying the data
> > predominates over the computation time, maybe you didn't need
> > parallelization in the first place.
> >
> > > 2) Removing fork would break existing workflows
> >
> > I don't see why mclapply could not be rewritten using PSOCK clusters.
> > And as a side effect, this would enable those workflows on Windows,
> > which doesn't support fork.
> >
> > > Even if replaced with something using the same interface (e.g., a
> > > function that automatically detects variables to export as in the
> > > amazing `future` package), the lack of copy-on-write functionality
> > > would cause scripts everywhere to break.
> >
> > To implement copy-on-write, Linux overcommits virtual memory, and this
> > is what causes scripts to break unexpectedly: everything works fine,
> > until you change a small unimportant bit and... boom, out of memory.
> > And in general, running forks in any GUI would cause things everywhere
> > to break.
> >
> > > A simple example illustrating these two points:
> > > `x <- 5e8; mclapply(1:24, sum, x, 8)`
> > >
> > > Using fork, `mclapply` takes 5 seconds.  Using "psock", `clusterApply`
> > > does not complete.
> >
> > I'm not sure how did you setup that, but it does complete. Or do you
> > mean that you ran out of memory? Then try replacing "x" with, e.g.,
> > "x+1" in your mclapply example and see what happens (hint: save your
> > work first).
> >
> > --
> > Iñaki Úcar
>
> __
> R-devel@r-project

Re: [Rd] Use of C++ in Packages

[Kevin Ushey]

I think it's also worth saying that some of these issues affect C code
as well; e.g. this is not safe:

FILE* f = fopen(...);
Rf_eval(...);
fclose(f);

whereas the C++ equivalent would likely handle closing of the file in
the destructor. In other words, I think many users just may not be
cognizant of the fact that most R APIs can longjmp, and what that
implies for cleanup of allocated resources. R_alloc() may help solve
the issue specifically for memory allocations, but for any library
interface that has a 'open' and 'close' step, the same sort of issue
will arise.

What I believe we should do, and what Rcpp has made steps towards, is
make it possible to interact with some subset of the R API safely from
C++ contexts. This has always been possible with e.g. R_ToplevelExec()
and R_ExecWithCleanup(), and now things are even better with
R_UnwindProtect(). In theory, as a prototype, an R package could
provide a 'safe' C++ interface to the R API using R_UnwindProtect()
and friends as appropriate, and client packages could import and link
to that package to gain access to the interface. Code generators (as
Rcpp Attributes does) can handle some of the pain in these interfaces,
so that users are mostly insulated from the nitty gritty details.

I agree that the content of Tomas's post is very helpful, especially
since I expect many R programmers who dip their toes into the C++
world are not aware of the caveats of talking to R from C++. However,
I don't think it's helpful to recommend "don't use C++"; rather, I
believe the question should be, "what can we do to make it possible to
easily and safely interact with R from C++?". Because, as I understand
it, all of the problems raised are solvable: either through a
well-defined C++ interface, or through better education.

I'll add my own opinion: writing correct C code is an incredibly
difficult task. C++, while obviously not perfect, makes things
substantially easier with tools like RAII, the STL, smart pointers,
and so on. And I strongly believe that C++ (with Rcpp) is still a
better choice than C for new users who want to interface with R from
compiled code.

tl;dr: I (and I think most others) just wish the summary had a more
positive outlook for the future of C++ with R.

Best,
Kevin

On Fri, Mar 29, 2019 at 10:16 AM Simon Urbanek
 wrote:
>
> Jim,
>
> I think the main point of Tomas' post was to alert R users to the fact that 
> there are very serious issues that you have to understand when interfacing R 
> from C++. Using C++ code from R is fine, in many cases you only want to 
> access R data, use some library or compute in C++ and return results. Such 
> use-cases are completely fine in C++ as they don't need to trigger the issues 
> mentioned and it should be made clear that it was not what Tomas' blog was 
> about.
>
> I agree with Tomas that it is safer to give an advice to not use C++ to call 
> R API since C++ may give a false impression that you don't need to know what 
> you're doing. Note that it is possible to avoid longjmps by using 
> R_ExecWithCleanup() which can catch any longjmps from the called function. So 
> if you know what you're doing you can make things work. I think the issue 
> here is not necessarily lack of tools, it is lack of knowledge - which is why 
> I think Tomas' post is so important.
>
> Cheers,
> Simon
>
>
> > On Mar 29, 2019, at 11:19 AM, Jim Hester  wrote:
> >
> > First, thank you to Tomas for writing his recent post[0] on the R
> > developer blog. It raised important issues in interfacing R's C API
> > and C++ code.
> >
> > However I do _not_ think the conclusion reached in the post is helpful
> >> don’t use C++ to interface with R
> >
> > There are now more than 1,600 packages on CRAN using C++, the time is
> > long past when that type of warning is going to be useful to the R
> > community.
> >
> > These same issues will also occur with any newer language (such as
> > Rust or Julia[1]) which uses RAII to manage resources and tries to
> > interface with R. It doesn't seem a productive way forward for R to
> > say it can't interface with these languages without first doing
> > expensive copies into an intermediate heap.
> >
> > The advice to avoid C++ is also antithetical to John Chambers vision
> > of first S and R as a interface language (from Extending R [2])
> >
> >> The *interface* principle has always been central to R and to S
> > before. An interface to subroutines was _the_ way to extend the first
> > version of S. Subroutine interfaces have continued to be central to R.
> >
> > The book also has extensive sections on both C++ (via Rcpp) and Julia,
> > so clearly John thinks these are legitimate ways to extend R.
> >
> > So if 'don't use C++' is not realistic and the current R API does not
> > allow safe use of C++ exceptions what are the alternatives?
> >
> > One thing we could do is look how this is handled in other languages
> > written in C which also use longjmp for errors.
> >
> > Lua is one example

Re: [Rd] Objectsize function visiting every element for alt-rep strings

[Kevin Ushey]

I think that object.size() is most commonly used to answer the question,
"what R objects are consuming the most memory currently in my R session?"
and for that reason I think returning the size of the internal
representations of objects (for e.g. ALTREP objects; unevaluated promises)
is the right default behavior.

I also agree it would be worth considering adding arguments that control
how object.size() is computed for different kinds of R objects, since users
might want to use object.size() to answer different types of questions.

All that said, if the ultimate goal here is to avoid having RStudio
materialize ALTREP objects in the background, then perhaps that change
should happen in RStudio :-)

Best,
Kevin

On Tue, Jan 22, 2019 at 8:21 AM Tierney, Luke 
wrote:

> On Mon, 21 Jan 2019, Martin Maechler wrote:
>
> >> Travers Ching
> >> on Tue, 15 Jan 2019 12:50:45 -0800 writes:
> >
> >> I have a toy alt-rep string package that generates
> >> randomly seeded strings.  example: library(altstringisode)
> >> x <- altrandomStrings(1e8) head(x) [1]
> >> "2PN0bdwPY7CA8M06zVKEkhHgZVgtV1"
> >> "5PN2qmWqBlQ9wQj99nsQzldVI5ZuGX" ... etc object.size(1e8)
> >
> >> Object.size will call the set_altstring_Elt_method for
> >> every single element, materializing (slowly) every element
> >> of the vector.  This is a problem mostly in R-studio since
> >> object.size is called automatically, defeating the purpose
> >> of alt-rep.
>
> There is no sensible way in general to figure out how large the
> strings would be without computing them. There might be specifically
> for a deferred sequence conversion but it would require a fair bit of
> effort to figure out that would be better spent elsewhere.
>
> I've never been a big fan of object.size since what it is trying to
> compute isn't very well defined in the context of sharing and possible
> internal state changes (even before ALTREP byte code compilation could
> change the internals of a function [which object.size sees] and
> assigning into environments or evaluating promises can change
> environments [which object.size ignores]). The issue is not unlike the
> one faced by identical(), which has a bunch of options for the
> different ways objects can be identical, and might need even more.
>
> We could in general have object.size for and ALTREP return the
> object.size results of the current internal representation, but that
> might not always be appropriate. Again, what object.size is trying to
> compute isn't very well defined.
>
> RStudio does seem to call object.size on every assignment to
> .GlobalEnv. That might be worth revisiting.
>
>
> Best,
>
> luke
>
> >
> > Hmm.  But still, the idea had been that object.size()  *shuld*
> > return the size of the "de-ALTREP'ed" object *but* should not
> > de-ALTREP it.
> > That's what happens for integers, but indeed fails to happen for
> > such as.character(.)ed integers.
> >
> > From my eRum presentation (which took from the official ALTREP
> documentation
> > https://svn.r-project.org/R/branches/ALTREP/ALTREP.html ) :
> >
> >  > x <- 1:1e15
> >  > object.size(x) # 8000'000'000'000'048 bytes : 8000 TBytes -- ok, not
> really
> >  8048 bytes
> >  > is.unsorted(x) # FALSE : i.e., R's *knows* it is sorted
> >  [1] FALSE
> >  > xs <- sort(x)  #
> >  > .Internal(inspect(x))
> >  @80255f8 14 REALSXP g0c0 [NAM(7)]  1 : 1000 (compact)
> >  >
> >
> >  > cx <- as.character(x)
> >  > .Internal(inspect(cx))
> >  @80485d8 16 STRSXP g0c0 [NAM(1)]   
> >@80255f8 14 REALSXP g1c0 [MARK,NAM(7)]  1 : 1000 (compact)
> >  > system.time( print(object.size(x)), gc=FALSE)
> >  8048 bytes
> > user  system elapsed
> >0.000   0.000   0.001
> >  > system.time( print(object.size(cx)), gc=FALSE)
> >  Error: cannot allocate vector of size 8388608.0 Gb
> >  Timing stopped at: 11.43 0 11.46
> >  >
> >
> > One could consider it a bug that object.size(cx) is indeed
> > inspecting every string, i.e., accessing cx[i] for all i.
> > Note that it is *not*  deALTREPing cx  itself :
> >
> >> x <- 1:1e6
> >> cx <- as.character(x)
> >> .Internal(inspect(cx))
> >
> > @7f5b1a0 16 STRSXP g0c0 [NAM(1)]   
> >  @7f5adb0 13 INTSXP g0c0 [NAM(7)]  1 : 100 (compact)
> >> system.time( print(object.size(cx)), gc=FALSE)
> > 6448 bytes
> >   user  system elapsed
> >  0.369   0.005   0.374
> >> .Internal(inspect(cx))
> > @7f5b1a0 16 STRSXP g0c0 [NAM(7)]   
> >  @7f5adb0 13 INTSXP g0c0 [NAM(7)]  1 : 100 (compact)
> >>
> >
> >> Is there a way to avoid the problem of forced
> >> materialization in rstudio?
> >
> >> PS: Is there a way to tell if a post has been received by
> >> the mailing list?  How long does it take to show up in the
> >> archives?
> >
> > [ that (waiting time) distribution is quite right skewed... I'd
> >  guess it's median to be less than 10 minutes... but we had
> >  artificially delayed it somewhat in the past to fight
> >  s

Re: [Rd] methods(class = class()) - improve for |cl.| > 1 ?

[Kevin Ushey]

I think this would be a good change. I think most users use the
'methods(class = <...>)' function to answer the question, "what
methods can I call on objects with these classes?", and in that
context I think it would be sensible for the function to accept more
than one class.

Kevin

On Wed, Oct 17, 2018 at 7:15 AM Martin Maechler
 wrote:
>
> With new "strict" settings in R-devel, the following glm() example
>
> > data(anorexia, package = "MASS")
> > fm <- glm(Postwt ~ Prewt + Treat + offset(Prewt), family = gaussian, data = 
> > anorexia)
> > methods(class = class(fm))
> Warning in grep(name, row.names(info)) :
>   argument 'pattern' has length > 1 and only the first element will be used
> Warning in gsub(name, "", row.names(info)) :
>   argument 'pattern' has length > 1 and only the first element will be used
> Warning in grep(pattern, all.names, value = TRUE) :
>   argument 'pattern' has length > 1 and only the first element will be used
> Warning in grep(pattern, all.names, value = TRUE) :
>   argument 'pattern' has length > 1 and only the first element will be used
> ...
> ...
> ...
> [ca. 20 lines of warnings]
>
> and then shows the "glm" methods, but not "lm" ones.
>
> This is not a bug strictly, as  ?methods says that
>
>class: a symbol or character string naming a class: only used if
>   ‘generic.function’ is not supplied.
>
> and so the use of
>
>methods(class = class())
>
> is a user error when  class()  is of length > 1.
>
> In the case of e.g. a randomForest() result, we also get 25
> warnings, i.e. 50 lines, but then
>
> --->>>  no methods found
>
> because
>
>> class(rf.fit)
>[1] "randomForest.formula" "randomForest"
>
> and no methods are  defined for "randomForest.formula".
>
> ---
>
> Of course, all this works fine with S4 classes:  There the full
> inheritance is used and all methods are found.
>
> Still, would it make sense to improve the underlying .S3methods() ?
>
> I assume it will break *some* overzealous package checks out
> there when .S3methods() and hence methods() would return *more*
> in such case.
>
> Comments?
>
> --
> Martin Maechler
> ETH Zurich  and  R Core
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Bug when calling system/system2 (and request for Bugzilla account)

[Kevin Ushey]

FWIW I can reproduce on macOS with R 3.5.1. A smaller example:

system2("ls", timeout = 5); x <- sample(1:1E8)

If I try to interrupt R while that sample call is running, R itself is closed.

Best,
Kevin

On Fri, Sep 14, 2018 at 10:53 AM Emil Bode  wrote:
>
> I hope it's not too specific in my setup...
> I've tried with system2 added on the first line, so:
>
> Example.R:
> system2('ls', timeout=5)
> cat('Start non-interruptable functions\n')
> sample_a <- sample(1:1e7)
> sample_b <- sample(1:2e7)
> matching <- match(sample_a, sample_b)
> cat('Finished\n')
> Sys.sleep(10)
>
> And in terminal/bash:
> R --vanilla
> source('Example.R')
> Send ^C between the messages (Start...  until Finished)
>
> Or if you have a more powerful CPU you can increase the samples, the exact 
> code doesn't matter very much.
> As soon as you restart and source again with the system2 call outcommented, 
> the behaviour is different, there is a pause, and your return to the R-prompt.
>
> Best, Emil
>
>
>
> On 14/09/2018, 17:39, "luke-tier...@uiowa.edu"  
> wrote:
>
> I can't reproduce this. Can you be more precise: exactly where are you
> putting the system2 call and exactly where are you sending the
> interrupt signal with ^C?
>
> Best,
>
> luke
>
> On Fri, 14 Sep 2018, Emil Bode wrote:
>
> > Hi all,
> >
> > I found some strange behaviour, which I think is a bug. Could someone 
> make an account for me on Bugzilla or pass on my report?
> >
> > The problem:
> > When pressing Ctrl-C when a file is sourced in R, run from Terminal 
> (macOS), sometimes the entire session is ended right away, while I just want 
> to stop the script. This is the case when I press Ctrl-C while some functions 
> are running that don’t catch the interrupt. However, the behaviour is 
> different whether I’m in a clean session (in which case some time is allowed 
> to pass, so that when the function returns the script can be interrupted), or 
> whether I have called base::system() or system2() with timeout other than 0.
> >
> > Reproducible example:
> > cat('Start non-interruptable functions\n')
> > sample_a <- sample(1:1e7)
> > sample_b <- sample(1:2e7)
> > matching <- match(sample_a, sample_b)
> > cat('Finished\n')
> > Sys.sleep(10)
> >
> > Observed behaviour:
> > In a clean session, when I hit Ctrl-C during the execution of match, 
> there is a delay, and as soon as Sys.sleep() is invoked, the script is 
> interrupted, I get back my R “>”-prompt (unless options(error=…) is set)
> > But If I add the line system2("ls", timeout = 5), or something similar, 
> when I try to break during the first part of the script, my Rsession ends, I 
> get thrown back to my terminal-prompt.
> >
> > Desired behaviour:
> > The best setup would probably be if Ctrl-C would always try to break 
> from the sourced file, and only if that doesn’t success in n seconds, break 
> the rsession altogether, ideally with a customizable option. But maybe that’s 
> too hard, so maybe the most pragmatic would be to have 2 hotkeys: one to 
> break from a hanging/broken rsession, and one to gently try to break from a 
> script. But at least I think it should be:
> >
> > Expected behaviour:
> > Consistent behaviour for Ctrl-C: either trying to break the script, or 
> end the session altogether.
> >
> > Some observations:
> >
> >  *   I can still break cleanly during the Sys.sleep(). But for larger 
> scripts, it is largely a matter of luck if I hit Ctrl-C during the right 
> moment.
> >  *   I don’t notice any difference between using system or system2, or 
> any of the arguments other than timeout provided
> >  *   I don’t notice any difference whether the timeout is actually 
> exhausted or not.
> >  *   Later calls to system/system2 don’t change anything (i.e. later 
> calling system(…, timeout=0) does not revert back to the old situation)
> >
> > My setup:
> > R 3.5.1 (Feather Spray), run with –vanilla option
> > GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
> > macOS High Sierra 10.13.6
> >
> > Best regards,
> > Emil Bode
> >
> > Data-analyst
> >
> > +31 6 43 83 89 33
> > emil.b...@dans.knaw.nl
> >
> > DANS: Netherlands Institute for Permanent Access to Digital Research 
> Resources
> > Anna van Saksenlaan 51 | 2593 HW Den Haag | +31 70 349 44 50 | 
> i...@dans.knaw.nl | 
> dans.knaw.nl
> > DANS is an institute of the Dutch Academy KNAW and 
> funding organisation NWO.
> >
> >   [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
>

Re: [Rd] True length - length(unclass(x)) - without having to call unclass()?

[Kevin Ushey]

More generally, I think one of the issues is that R is not yet able to
decrement a reference count (or mark a 'shared' data object as
'unshared' after it knows only one binding to it exists). This means
passing variables to R closures will mark that object as shared:

x <- list()
.Internal(inspect(x))  # NAM(1)
identity(x)
.Internal(inspect(x))  # NAM(3)

I think for this reason users often resort to 'hacks' that involve
directly setting attributes on the object, since they 'know' only one
reference to a particular object exists. I'm not sure if this really
is 'safe', though -- likely not given potential future optimizations
to R, as Tomas has alluded to.

I think true reference counting has been implemented in the R sources,
but the switch has not yet been flipped to enable that by default.
Hopefully having that will make cases like the above work as expected?

Thanks,
Kevin

On Wed, Sep 5, 2018 at 2:19 AM Iñaki Ucar  wrote:
>
> The bottomline here is that one can always call a base method,
> inexpensively and without modifying the object, in, let's say,
> *formal* OOP languages. In R, this is not possible in general. It
> would be possible if there was always a foo.default, but primitives
> use internal dispatch.
>
> I was wondering whether it would be possible to provide a super(x, n)
> function which simply causes the dispatching system to avoid "n"
> classes in the hierarchy, so that:
>
> > x <- structure(list(), class=c("foo", "bar"))
> > length(super(x, 0)) # looks for a length.foo
> > length(super(x, 1)) # looks for a length.bar
> > length(super(x, 2)) # calls the default
> > length(super(x, Inf)) # calls the default
>
> Iñaki
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Output mis-encoded on Windows w/ RGui 3.5.1 in strange case

[Kevin Ushey]

Thank you for the quick fix! I could've sworn the 'save()' dance was a
necessary part of the reproducible example, but evidently not ...
On Wed, Jul 18, 2018 at 6:38 AM Tomas Kalibera  wrote:
>
> Fixed in R-devel and R-patched,
> Tomas
>
> On 07/18/2018 12:03 PM, Tomas Kalibera wrote:
>
> Thanks, I can now reproduce and it is a bug that is easy to fix, I will do so 
> shortly.
>
> Fyi it can be reproduced simply by running these two lines in Rgui:
>
> list()
> encodeString("apple")
>
> Best
> Tomas
>
> On 07/17/2018 05:16 PM, Kevin Ushey wrote:
>
> Sorry, I should have been more clear -- if I write the contents of
> that script to a file called 'encoding.R' and source that, then I see
> the reported behavior.
>
> Here's something standalone that you should hopefully be able to copy
> + paste into RGui to reproduce:
>
> code <- '
>x <- 1
>print(list())
>save(x, file = tempfile())
>output <- encodeString("apple")
>print(output)
> '
>
> file <- tempfile(fileext = ".R")
> writeLines(code, con = file)
> source(file)
>
> When I run this, I see:
>
> code <- '
>
> +x <- 1
> +print(list())
> +save(x, file = tempfile())
> +output <- encodeString("apple")
> +print(output)
> + '
>
> file <- tempfile(fileext = ".R")
> writeLines(code, con = file)
> source(file)
>
> list()
> [1] "\002ÿþapple\003ÿþ"
>
> This is with today's R-devel:
>
> sessionInfo()
>
> R Under development (unstable) (2018-07-16 r74967)
> Platform: x86_64-w64-mingw32/x64 (64-bit)
> Running under: Windows 10 x64 (build 17134)
>
> Matrix products: default
>
> locale:
> [1] LC_COLLATE=English_United States.1252
> [2] LC_CTYPE=English_United States.1252
> [3] LC_MONETARY=English_United States.1252
> [4] LC_NUMERIC=C
> [5] LC_TIME=English_United States.1252
>
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods   base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.6.0
>
> I realize the example looks incomplete, but it seems like each step is
> required to reproduce the strange behavior:
>
>1) You need to print an empty list,
>2) You need to invoke save() after printing that empty list,
>3) Then, attempts to call encodeString() will produce the strange output.
>
> For what it's worth, it may be related to a behavior I'm seeing where
> the first name printed for an R list is quoted with backticks even
> when not necessary:
>
> list(x = 1, y = 2)
>
> $`x`
> [1] 1
>
> $y
> [1] 2
>
> Thanks,
> Kevin
>
> On Tue, Jul 17, 2018 at 6:12 AM Tomas Kalibera  
> wrote:
>
> Hi Kevin,
>
> the extra bytes you are seeing are escapes for UTF-8 strings used in
> input to RGui console. Recently ascii strings are converted to UTF-8 so
> you would get these escapes for ascii strings now as well. RGui
> understands these escapes and converts from UTF-8 to wide characters
> before printing on Windows. The escapes should not be used unless
> printing to RGui console.
>
> I suppose you managed to leak the escapes but I cannot reproduce, the
> example you sent seems incomplete ("x" not used, not clear what
> encoding.R is, not clear where the encodeString is run) and none of the
> variations I ran leaked the escapes on R-devel. Please clarify the
> example if you believe it is a bug. Please also use current R-devel
> (I've relatively recently fixed a bug in decoding these escaped strings,
> perhaps unlikely, but not impossible it could be related).
>
> Best
> Tomas
>
> On 07/16/2018 10:01 PM, Kevin Ushey wrote:
>
> Given the following R script:
>
> x <- 1
> print(list())
> save(x, file = tempfile())
> output <- encodeString("apple")
> print(output)
>
> If I source this script from RGui on Windows, I see the output:
>
> > source("encoding.R")
> list()
> [1] "\002ÿþapple\003ÿþ"
>
> That is, it's as though R has injected what looks like byte order
> marks around the encoded string:
>
> > charToRaw(output)
>  [1] 02 ff fe 61 70 70 6c 65 03 ff fe
>
> FWIW I see the same output in R-patched and R-devel. Any idea what
> might be going on? For what it's worth, I don't see the same issue
> with R as run from the terminal.
>
> Thanks,
> Kevin
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Output mis-encoded on Windows w/ RGui 3.5.1 in strange case

[Kevin Ushey]

Sorry, I should have been more clear -- if I write the contents of
that script to a file called 'encoding.R' and source that, then I see
the reported behavior.

Here's something standalone that you should hopefully be able to copy
+ paste into RGui to reproduce:

code <- '
   x <- 1
   print(list())
   save(x, file = tempfile())
   output <- encodeString("apple")
   print(output)
'

file <- tempfile(fileext = ".R")
writeLines(code, con = file)
source(file)

When I run this, I see:

> code <- '
+x <- 1
+print(list())
+save(x, file = tempfile())
+output <- encodeString("apple")
+print(output)
+ '
>
> file <- tempfile(fileext = ".R")
> writeLines(code, con = file)
> source(file)
list()
[1] "\002ÿþapple\003ÿþ"

This is with today's R-devel:

> sessionInfo()
R Under development (unstable) (2018-07-16 r74967)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.0

I realize the example looks incomplete, but it seems like each step is
required to reproduce the strange behavior:

   1) You need to print an empty list,
   2) You need to invoke save() after printing that empty list,
   3) Then, attempts to call encodeString() will produce the strange output.

For what it's worth, it may be related to a behavior I'm seeing where
the first name printed for an R list is quoted with backticks even
when not necessary:

> list(x = 1, y = 2)
$`x`
[1] 1

$y
[1] 2

Thanks,
Kevin

On Tue, Jul 17, 2018 at 6:12 AM Tomas Kalibera  wrote:
>
> Hi Kevin,
>
> the extra bytes you are seeing are escapes for UTF-8 strings used in
> input to RGui console. Recently ascii strings are converted to UTF-8 so
> you would get these escapes for ascii strings now as well. RGui
> understands these escapes and converts from UTF-8 to wide characters
> before printing on Windows. The escapes should not be used unless
> printing to RGui console.
>
> I suppose you managed to leak the escapes but I cannot reproduce, the
> example you sent seems incomplete ("x" not used, not clear what
> encoding.R is, not clear where the encodeString is run) and none of the
> variations I ran leaked the escapes on R-devel. Please clarify the
> example if you believe it is a bug. Please also use current R-devel
> (I've relatively recently fixed a bug in decoding these escaped strings,
> perhaps unlikely, but not impossible it could be related).
>
> Best
> Tomas
>
> On 07/16/2018 10:01 PM, Kevin Ushey wrote:
> > Given the following R script:
> >
> > x <- 1
> > print(list())
> > save(x, file = tempfile())
> > output <- encodeString("apple")
> > print(output)
> >
> > If I source this script from RGui on Windows, I see the output:
> >
> > > source("encoding.R")
> > list()
> > [1] "\002ÿþapple\003ÿþ"
> >
> > That is, it's as though R has injected what looks like byte order
> > marks around the encoded string:
> >
> > > charToRaw(output)
> >  [1] 02 ff fe 61 70 70 6c 65 03 ff fe
> >
> > FWIW I see the same output in R-patched and R-devel. Any idea what
> > might be going on? For what it's worth, I don't see the same issue
> > with R as run from the terminal.
> >
> > Thanks,
> > Kevin
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Output mis-encoded on Windows w/ RGui 3.5.1 in strange case

[Kevin Ushey]

Given the following R script:

   x <- 1
   print(list())
   save(x, file = tempfile())
   output <- encodeString("apple")
   print(output)

If I source this script from RGui on Windows, I see the output:

   > source("encoding.R")
   list()
   [1] "\002ÿþapple\003ÿþ"

That is, it's as though R has injected what looks like byte order
marks around the encoded string:

   > charToRaw(output)
[1] 02 ff fe 61 70 70 6c 65 03 ff fe

FWIW I see the same output in R-patched and R-devel. Any idea what
might be going on? For what it's worth, I don't see the same issue
with R as run from the terminal.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] writeLines argument useBytes = TRUE still making conversions

[Kevin Ushey]

Of course, right after writing this e-mail I tested on my Windows
machine and did not see what I expected:

> charToRaw(before)
[1] c3 a9
> charToRaw(after)
[1] e9

so obviously I'm misunderstanding something as well.

Best,
Kevin

On Sat, Feb 17, 2018 at 2:19 PM, Kevin Ushey  wrote:
> From my understanding, translation is implied in this line of ?file (from the
> Encoding section):
>
> The encoding of the input/output stream of a connection can be specified
> by name in the same way as it would be given to iconv: see that help page
> for how to find out what encoding names are recognized on your platform.
> Additionally, "" and "native.enc" both mean the ‘native’ encoding, that is
> the internal encoding of the current locale and hence no translation is
> done.
>
> This is also hinted at in the documentation in ?readLines for its 'encoding'
> argument, which has a different semantic meaning from the 'encoding' argument
> as used with R connections:
>
> encoding to be assumed for input strings. It is used to mark character
> strings as known to be in Latin-1 or UTF-8: it is not used to re-encode
> the input. To do the latter, specify the encoding as part of the
> connection con or via options(encoding=): see the examples.
>
> It might be useful to augment the documentation in ?file with something like:
>
> The 'encoding' argument is used to request the translation of strings when
> writing to a connection.
>
> and, perhaps to further drive home the point about not translating when
> encoding = "native.enc":
>
> Note that R will not attempt translation of strings when encoding is
> either "" or "native.enc" (the default, as per getOption("encoding")).
> This implies that attempting to write, for example, UTF-8 encoded content
> to a connection opened using "native.enc" will retain its original UTF-8
> encoding -- it will not be translated.
>
> It is a bit surprising that 'native.enc' means "do not translate" rather than
> "attempt translation to the encoding associated with the current locale", but
> those are the semantics and they are not bound to change.
>
> This is the code I used to convince myself of that case:
>
> conn <- file(tempfile(), encoding = "native.enc", open = "w+")
>
> before <- iconv('é', to = "UTF-8")
> cat(before, file = conn, sep = "\n")
> after <- readLines(conn)
>
> charToRaw(before)
> charToRaw(after)
>
> with output:
>
> > charToRaw(before)
> [1] c3 a9
> > charToRaw(after)
> [1] c3 a9
>
> Best,
> Kevin
>
>
> On Thu, Feb 15, 2018 at 9:16 AM, Ista Zahn  wrote:
>> On Thu, Feb 15, 2018 at 11:19 AM, Kevin Ushey  wrote:
>>> I suspect your UTF-8 string is being stripped of its encoding before
>>> write, and so assumed to be in the system native encoding, and then
>>> re-encoded as UTF-8 when written to the file. You can see something
>>> similar with:
>>>
>>> > tmp <- 'é'
>>> > tmp <- iconv(tmp, to = 'UTF-8')
>>> > Encoding(tmp) <- "unknown"
>>> > charToRaw(iconv(tmp, to = "UTF-8"))
>>> [1] c3 83 c2 a9
>>>
>>> It's worth saying that:
>>>
>>> file(..., encoding = "UTF-8")
>>>
>>> means "attempt to re-encode strings as UTF-8 when writing to this
>>> file". However, if you already know your text is UTF-8, then you
>>> likely want to avoid opening a connection that might attempt to
>>> re-encode the input. Conversely (assuming I'm understanding the
>>> documentation correctly)
>>>
>>> file(..., encoding = "native.enc")
>>>
>>> means "assume that strings are in the native encoding, and hence
>>> translation is unnecessary". Note that it does not mean "attempt to
>>> translate strings to the native encoding".
>>
>> If all that is true I think ?file needs some attention. I've read it
>> several times now and I just don't see how it can be interpreted as
>> you've described it.
>>
>> Best,
>> Ista
>>
>>>
>>> Also note that writeLines(..., useBytes = FALSE) will explicitly
>>> translate to the current encoding before sending bytes to the
>>> requested connection. In other words, there are two locations where
>>> tran

Re: [Rd] writeLines argument useBytes = TRUE still making conversions

[Kevin Ushey]

>From my understanding, translation is implied in this line of ?file (from the
Encoding section):

The encoding of the input/output stream of a connection can be specified
by name in the same way as it would be given to iconv: see that help page
for how to find out what encoding names are recognized on your platform.
Additionally, "" and "native.enc" both mean the ‘native’ encoding, that is
the internal encoding of the current locale and hence no translation is
done.

This is also hinted at in the documentation in ?readLines for its 'encoding'
argument, which has a different semantic meaning from the 'encoding' argument
as used with R connections:

encoding to be assumed for input strings. It is used to mark character
strings as known to be in Latin-1 or UTF-8: it is not used to re-encode
the input. To do the latter, specify the encoding as part of the
connection con or via options(encoding=): see the examples.

It might be useful to augment the documentation in ?file with something like:

The 'encoding' argument is used to request the translation of strings when
writing to a connection.

and, perhaps to further drive home the point about not translating when
encoding = "native.enc":

Note that R will not attempt translation of strings when encoding is
either "" or "native.enc" (the default, as per getOption("encoding")).
This implies that attempting to write, for example, UTF-8 encoded content
to a connection opened using "native.enc" will retain its original UTF-8
encoding -- it will not be translated.

It is a bit surprising that 'native.enc' means "do not translate" rather than
"attempt translation to the encoding associated with the current locale", but
those are the semantics and they are not bound to change.

This is the code I used to convince myself of that case:

conn <- file(tempfile(), encoding = "native.enc", open = "w+")

before <- iconv('é', to = "UTF-8")
cat(before, file = conn, sep = "\n")
after <- readLines(conn)

charToRaw(before)
charToRaw(after)

with output:

> charToRaw(before)
[1] c3 a9
> charToRaw(after)
[1] c3 a9

Best,
Kevin


On Thu, Feb 15, 2018 at 9:16 AM, Ista Zahn  wrote:
> On Thu, Feb 15, 2018 at 11:19 AM, Kevin Ushey  wrote:
>> I suspect your UTF-8 string is being stripped of its encoding before
>> write, and so assumed to be in the system native encoding, and then
>> re-encoded as UTF-8 when written to the file. You can see something
>> similar with:
>>
>> > tmp <- 'é'
>> > tmp <- iconv(tmp, to = 'UTF-8')
>> > Encoding(tmp) <- "unknown"
>> > charToRaw(iconv(tmp, to = "UTF-8"))
>> [1] c3 83 c2 a9
>>
>> It's worth saying that:
>>
>> file(..., encoding = "UTF-8")
>>
>> means "attempt to re-encode strings as UTF-8 when writing to this
>> file". However, if you already know your text is UTF-8, then you
>> likely want to avoid opening a connection that might attempt to
>> re-encode the input. Conversely (assuming I'm understanding the
>> documentation correctly)
>>
>> file(..., encoding = "native.enc")
>>
>> means "assume that strings are in the native encoding, and hence
>> translation is unnecessary". Note that it does not mean "attempt to
>> translate strings to the native encoding".
>
> If all that is true I think ?file needs some attention. I've read it
> several times now and I just don't see how it can be interpreted as
> you've described it.
>
> Best,
> Ista
>
>>
>> Also note that writeLines(..., useBytes = FALSE) will explicitly
>> translate to the current encoding before sending bytes to the
>> requested connection. In other words, there are two locations where
>> translation might occur in your example:
>>
>>1) In the call to writeLines(),
>>2) When characters are passed to the connection.
>>
>> In your case, it sounds like translation should be suppressed at both steps.
>>
>> I think this is documented correctly in ?writeLines (and also the
>> Encoding section of ?file), but the behavior may feel unfamiliar at
>> first glance.
>>
>> Kevin
>>
>> On Wed, Feb 14, 2018 at 11:36 PM, Davor Josipovic  wrote:
>>>
>>> I think this behavior is inconsistent with the documentation:
>>>
>>>   tmp <- 'é'
>>>   tm

Re: [Rd] writeLines argument useBytes = TRUE still making conversions

[Kevin Ushey]

I suspect your UTF-8 string is being stripped of its encoding before
write, and so assumed to be in the system native encoding, and then
re-encoded as UTF-8 when written to the file. You can see something
similar with:

> tmp <- 'é'
> tmp <- iconv(tmp, to = 'UTF-8')
> Encoding(tmp) <- "unknown"
> charToRaw(iconv(tmp, to = "UTF-8"))
[1] c3 83 c2 a9

It's worth saying that:

file(..., encoding = "UTF-8")

means "attempt to re-encode strings as UTF-8 when writing to this
file". However, if you already know your text is UTF-8, then you
likely want to avoid opening a connection that might attempt to
re-encode the input. Conversely (assuming I'm understanding the
documentation correctly)

file(..., encoding = "native.enc")

means "assume that strings are in the native encoding, and hence
translation is unnecessary". Note that it does not mean "attempt to
translate strings to the native encoding".

Also note that writeLines(..., useBytes = FALSE) will explicitly
translate to the current encoding before sending bytes to the
requested connection. In other words, there are two locations where
translation might occur in your example:

   1) In the call to writeLines(),
   2) When characters are passed to the connection.

In your case, it sounds like translation should be suppressed at both steps.

I think this is documented correctly in ?writeLines (and also the
Encoding section of ?file), but the behavior may feel unfamiliar at
first glance.

Kevin

On Wed, Feb 14, 2018 at 11:36 PM, Davor Josipovic  wrote:
>
> I think this behavior is inconsistent with the documentation:
>
>   tmp <- 'é'
>   tmp <- iconv(tmp, to = 'UTF-8')
>   print(Encoding(tmp))
>   print(charToRaw(tmp))
>   tmpfilepath <- tempfile()
>   writeLines(tmp, con = file(tmpfilepath, encoding = 'UTF-8'), useBytes = 
> TRUE)
>
> [1] "UTF-8"
> [1] c3 a9
>
> Raw text as hex: c3 83 c2 a9
>
> If I switch to useBytes = FALSE, then the variable is written correctly as  
> c3 a9.
>
> Any thoughts? This behavior is related to this issue: 
> https://github.com/yihui/knitr/issues/1509
>
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Bug: Issues on Windows with SFN disabled

[Kevin Ushey]

For what it's worth, the Windows installers for other programming
language runtimes often install outside of Program Files, so at least
there is 'prior art' to motivate having R install directly into the
root of the home drive:

- ActiveState Perl installs directly C:/Perl;
- Python installs (when installing for all users) into C:/Python$VERSION;
- The Ruby installers at https://rubyinstaller.org/ default to the
root home drive.

So I (as Dirk said earlier) would also be in favor of having R install
directly to the root of the home drive, with e.g. C:/R/R-x.y.z being
the default install location.

Best,
Kevin


On Fri, Dec 8, 2017 at 4:14 AM, Tomas Kalibera  wrote:
>
> On 12/07/2017 06:15 PM, Dirk Eddelbuettel wrote:
>>
>> On 7 December 2017 at 17:56, Tomas Kalibera wrote:
>> |
>> | An update on this. Writing R Extensions does not recommend to have a
>> | space character in R_HOME. This means that on Windows one either should
>> | have SFN enabled (which is still the common case), or install into a
>> | directory that does not have a space in its name (so specifically not
>> | into "Program Files"). This recommendation unfortunately needs to stay
>> | for now.
>> |
>> | WRE recommends that Makefiles are written to be robust against space
>> | characters inside R_HOME. All path names passed from a Makefile to the
>> | shell should be quoted at least if they include R_HOME. Make "include"
>> | directives should not be used on path names that are derived from
>> | R_HOME, but one should instead use the "-f" option multiple times when
>> | recursively invoking make. Maintainers of packages that use "include"
>> | with R_HOME have been notified. Unfortunately, the number of packages
>> | that do not quote pathnames with R_HOME in Makefiles is rather large, so
>> | fixing will take some time.
>> |
>> | Currently, R-devel should build fine on Windows with R_HOME including
>> | space, including all base and recommended packages, and tests for these
>> | packages should pass even though this is not regularly tested. If you
>> | find a case when this does not work, please submit a bug report.
>>
>> Why does the Windows installer default to using a directory with spaces?
>
> It's a convention on Windows and I guess there may be problems with 
> permissions on other directories. My hope is we can make R work reliably 
> without SFN just in time before SFN become disabled by default, after all, 
> quoting pathnames in Makefiles (or shell scripts for that matter) is a good 
> practice anyway and avoiding "include" is not a big problem as very few 
> packages are affected.
>
> But thanks for opening this and I am happy for insights from any Windows 
> experts on the issue. I would not want to violate the convention for all 
> users when just few of them have SFN disabled, and as I hope this will be 
> fixed on R/packages side, but maybe the installer could at least detect the 
> problem (when "Program Files" or another specified target directory did not 
> have a short name). Or perhaps also suggest a different default. Certainly R 
> could print a warning when it starts.
>
> Tomas
>>
>> Related (but moderately more advanced), why does R still install "everything"
>> under one (versioned) directory so that uninformed users on upgrade "miss"
>> all previously installed packages?
>>
>> Why not (with space for exposition here, imagine s/ // everywhere below)
>>
>>  $SOMEROOTDIR / R /
>>R-a.b.c/  # before
>>R-a.b.d/  # d > c, here
>>site-library/ # with .libPaths having this preset?
>>
>> I don't really care as I manage to work mostly / entirely on another OS, but
>> I just don't understand why we do not put two and two together. But I am
>> likely unaware of some salient issues.
>
>
>>
>> In any event, I appreciate the thankless work of those taking care of Windoze
>> (ie Tomas, Jeroen, Duncan (now ex-officio), Brian, ...)
>>
>> Dirk
>>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] unlicense

[Kevin Ushey]

The Free Software Foundation maintains a list of free and GPL-compatible
software licenses here:

https://www.gnu.org/licenses/license-list.en.html#Unlicense

It appears that Unlicense is considered a free and GPL-compatible license;
however, the page does suggest using CC0 instead (which is indeed a license
approved / recognized by CRAN). CC0 appears to be the primary license
recommended by the FSF for software intended for the public domain.

On Tue, Jan 17, 2017 at 8:32 PM, Karl Millar via R-devel <
r-devel@r-project.org> wrote:

> Unfortunately, our lawyers say that they can't give legal advice in
> this context.
>
> My question would be, what are people looking for that the MIT or
> 2-clause BSD license don't provide?  They're short, clear, widely
> accepted and very permissive.  Another possibility might be to
> dual-license packages with both an OSI-approved license and
> whatever-else-you-like, e.g.  'MIT | ', but IIUC
> there's a bunch more complexity there than just using an OSI-approved
> license.
>
> Karl
>
>
> On Tue, Jan 17, 2017 at 3:35 PM, Uwe Ligges
>  wrote:
> >
> >
> > On 18.01.2017 00:13, Karl Millar wrote:
> >>
> >> Please don't use 'Unlimited' or 'Unlimited + ...'.
> >>
> >> Google's lawyers don't recognize 'Unlimited' as being open-source, so
> >> our policy doesn't allow us to use such packages due to lack of an
> >> acceptable license.  To our lawyers, 'Unlimited + file LICENSE' means
> >> something very different than it presumably means to Uwe.
> >
> >
> >
> > Karl,
> >
> > thanks for this comment. What we like to hear now is a suggestion what
> the
> > maintainer is supposed to do to get what he aims at, as we already know
> that
> > "freeware" does not work at all and was hard enough to get to the
> > "Unlimited" options.
> >
> > We have many CRAN requests asking for what they should write for
> "freeware".
> > Can we get an opinion from your layers which standard license comes
> closest
> > to what these maintainers probably aim at and will work more or less
> > globally, i.e. not only in the US?
> >
> > Best,
> > Uwe
> >
> >
> >
> >
> >> Thanks,
> >>
> >> Karl
> >>
> >> On Sat, Jan 14, 2017 at 12:10 AM, Uwe Ligges
> >>  wrote:
> >>>
> >>> Dear all,
> >>>
> >>> from "Writing R Extensions":
> >>>
> >>> The string ‘Unlimited’, meaning that there are no restrictions on
> >>> distribution or use other than those imposed by relevant laws
> (including
> >>> copyright laws).
> >>>
> >>> If a package license restricts a base license (where permitted, e.g.,
> >>> using
> >>> GPL-3 or AGPL-3 with an attribution clause), the additional terms
> should
> >>> be
> >>> placed in file LICENSE (or LICENCE), and the string ‘+ file LICENSE’
> (or
> >>> ‘+
> >>> file LICENCE’, respectively) should be appended to the
> >>> corresponding individual license specification.
> >>> ...
> >>> Please note in particular that “Public domain” is not a valid license,
> >>> since
> >>> it is not recognized in some jurisdictions."
> >>>
> >>> So perhaps you aim for
> >>> License: Unlimited
> >>>
> >>> Best,
> >>> Uwe Ligges
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 14.01.2017 07:53, Deepayan Sarkar wrote:
> 
> 
>  On Sat, Jan 14, 2017 at 5:49 AM, Duncan Murdoch
>   wrote:
> >
> >
> > On 13/01/2017 3:21 PM, Charles Geyer wrote:
> >>
> >>
> >>
> >> I would like the unlicense (http://unlicense.org/) added to R
> >> licenses.  Does anyone else think that worthwhile?
> >>
> >
> > That's a question for you to answer, not to ask.  Who besides you
> > thinks
> > that it's a good license for open source software?
> >
> > If it is recognized by the OSF or FSF or some other authority as a
> FOSS
> > license, then CRAN would probably also recognize it.  If not, then
> CRAN
> > doesn't have the resources to evaluate it and so is unlikely to
> > recognize
> > it.
> 
> 
> 
>  Unlicense is listed in https://spdx.org/licenses/
> 
>  Debian does include software "licensed" like this, and seems to think
>  this is one way (not the only one) of declaring something to be
>  "public domain".  The first two examples I found:
> 
>  https://tracker.debian.org/media/packages/r/rasqal/copyright-0.9.29-1
> 
> 
>  https://tracker.debian.org/media/packages/w/wiredtiger/
> copyright-2.6.1%2Bds-1
> 
>  This follows the format explained in
> 
> 
>  https://www.debian.org/doc/packaging-manuals/copyright-
> format/1.0/#license-specification,
>  which does not explicitly include Unlicense, but does include CC0,
>  which AFAICT is meant to formally license something so that it is
>  equivalent to being in the public domain. R does include CC0 as a
>  shorthand (e.g., geoknife).
> 
>  https://www.debian.org/legal/licenses/ says that
> 
>  
> 
>  Licenses currently found in Debian main include:
> 
>  - ...
>  - ...
>  - public domain (n

Re: [Rd] require(..., quietly=TRUE) does not suppress warning

[Kevin Ushey]

IMHO the strongest argument for suppressing the warning message here is the
fact that

requireNamespace("foo", quietly = TRUE)

does not emit any warning message when the package 'foo' does not exist.

On Thu, Dec 8, 2016 at 12:51 PM, Dan Tenenbaum 
wrote:

> Well, I'm getting a warning (not an error) when the package doesn't exist.
> I interpreted "most often" to mean that suppressing warnings/errors is why
> you'd most often use this argument, as most packages don't emit startup
> messages.
>
> And technically there isn't a problem with attaching the package, since we
> don't even try to attach packages that don't exist.
>
> So yes, very careful parsing of the docs suggests that the behavior is
> correct, but it does seem to violate the 'spirit' of what a user might
> expect. I am pretty sure I have used the 'if (!require("pkg"))
> install.packages("pkg")' pattern before without seeing this warning, so I
> wondered if the behavior had changed, and that's what prompted me to write.
>
> I know I can squelch the warning by wrapping the require() in
> suppressWarnings().
>
> Dan
>
>
> - Original Message -
> > From: "John P. Nolan" 
> > To: "Dan Tenenbaum" , "R-devel" <
> r-devel@r-project.org>
> > Sent: Thursday, December 8, 2016 12:37:02 PM
> > Subject: RE: require(..., quietly=TRUE) does not suppress warning
>
> > Well, it says "most often" no errors/warnings are given, so it is not
> > contradicting the docs!   It looks like the person/team that coded
> require( )
> > decided you should get an error when the package doesn't exist.
> >
> > If you want a silent loading, consider
> > aaa <- try( library(foo,verbose=FALSE,quietly=TRUE),silent=TRUE)
> > and then check to see if aaa is of class "try-error" and check for
> failure
> >
> > John
> > ……..
> >
> > John P. Nolan
> > Math/Stat Dept., American University
> > Gray Hall, 4400 Massachusetts Ave, NW
> > Washington, DC 20016-8050
> > Phone: 202-885-3140
> > E-mail:  jpno...@american.edu
> > Web:   http://fs2.american.edu/jpnolan/www/
> >
> >
> >
> > -Original Message
> > From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Dan
> Tenenbaum
> > Sent: Thursday, December 8, 2016 2:43 PM
> > To: R-devel 
> > Subject: [Rd] require(..., quietly=TRUE) does not suppress warning
> >
> > Hi,
> >
> > The `quietly` argument of `require` is documented as follows:
> >
> > quietly: a logical.  If ‘TRUE’, no message confirming package
> >  attaching is printed, and most often, no errors/warnings are
> >  printed if package attaching fails.
> >
> > However:
> >
> >> require(foo, quietly=TRUE)
> > Warning message:
> > In library(package, lib.loc = lib.loc, character.only = TRUE,
> logical.return =
> > TRUE,  :
> >  there is no package called ‘foo’
> >
> > Am I misreading the docs or is R misbehaving?
> >
> >> sessionInfo()
> > R version 3.3.2 (2016-10-31)
> > Platform: x86_64-apple-darwin13.4.0 (64-bit) Running under: macOS Sierra
> 10.12.1
> >
> > locale:
> > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
> >
> > attached base packages:
> > [1] stats graphics  grDevices utils datasets  methods   base
> >
> > Dan
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] creating a long list puts R in a state where many things stop working

[Kevin Ushey]

For reference, running your code in a build of R-devel with sanitizers:

> x <- vector(mode="list", length=3e9)
memory.c:2747:27: runtime error: signed integer overflow: 2147483647 + 1
cannot be represented in type 'R_len_t' (aka 'int')
SUMMARY: AddressSanitizer: undefined-behavior memory.c:2747:27 in
ASAN:DEADLYSIGNAL
=
==50159==ERROR: AddressSanitizer: SEGV on unknown address 0x
(pc 0x00010b3452a7 bp 0x7fff5669d2b0 sp 0x7fff5669cd80 T0)
==50159==The signal is caused by a READ memory access.
==50159==Hint: address points to the zero page.
#0 0x10b3452a6 in Rf_allocVector3 memory.c:2748
#1 0x10afc13b1 in do_makevector builtin.c:789
#2 0x10b238868 in bcEval eval.c:6041
#3 0x10b2241b8 in Rf_eval eval.c:626
#4 0x10b2794d4 in Rf_applyClosure eval.c:1160
#5 0x10b224d16 in Rf_eval eval.c:742
#6 0x10b286591 in do_set eval.c:2227
#7 0x10b2246ae in Rf_eval eval.c:695
#8 0x10b3229ef in Rf_ReplIteration main.c:258
#9 0x10b327280 in R_ReplConsole main.c:308
#10 0x10b327087 in run_Rmainloop main.c:1059
#11 0x10955fea4 in main Rmain.c:29
#12 0x7fffcabf3254 in start (libdyld.dylib+0x5254)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV memory.c:2748 in Rf_allocVector3
==50159==ABORTING
Abort trap: 6

Looks like there's an index here that needs to be R_xlen_t?
https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2747

Declared at
https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2476
-- could this index safely become R_xlen_t?

Kevin

On Mon, Nov 14, 2016 at 10:10 PM, Hervé Pagès  wrote:

> Hi,
>
> After I create a long list e.g. with
>
>   x <- vector(mode="list", length=3e9)
>
> many bad things start to happen e.g. some things stop working with a
> spurious error message:
>
>   gc()
>   # Error in gc() :
>   #   long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m
> ain/memory.c:1137
>
>   x_lens <- lengths(x)
>   # Error in lengths(x) :
>   #  long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m
> ain/memory.c:1668
>
> But then some of them work again:
>
>   gc()
>   #  used (Mb) gc trigger(Mb)   max used(Mb)
>   # Ncells  57046  3.1 36800019.7 3518.7
>   # Vcells 138060  1.1 4320596678 32963.6 4500397915 34335.4
>
> while others still fail but now with a different error message:
>
>   x_lens <- lengths(x)
>   # Error: evaluation nested too deeply: infinite recursion /
> options(expressions=)?
>
> etc...
>
> The more I go, the more weird things I see so clearly my session
> got corrupted. Finally, and not too surprisingly, after playing a
> little bit more, my session eventually crashed.
>
> Thanks,
> H.
>
> > sessionInfo()
> R version 3.3.2 (2016-10-31)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 14.04.3 LTS
>
> locale:
>  [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
>  [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
>  [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
>  [7] LC_PAPER=en_US.UTF-8   LC_NAME=C
>  [9] LC_ADDRESS=C   LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods   base
>
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpa...@fredhutch.org
> Phone:  (206) 667-5791
> Fax:(206) 667-1319
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] improve 'package not installed' load errors?

[Kevin Ushey]

Hi R-devel,

One of the more common issues that new R users see, and become stumped
by, is error messages during package load of the form:

> library(ggplot2)
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()),
versionCheck = vI[[j]]) :
  there is no package called 'Rcpp'
Error: package or namespace load failed for 'ggplot2'

Typically, error messages of this form are caused simply by one or
more dependent packages (in this case, 'Rcpp') not being installed or
available on the current library paths. (A side question, which I do
not know the answer to, is how users get themselves into this state.)

I believe it would be helpful for new users if the error message
reported here was a bit more direct, e.g.

> library(ggplot2)
Error: 'ggplot2' depends on package 'Rcpp', but 'Rcpp' is not installed
consider installing 'Rcpp' with install.packages("Rcpp")

In other words, it might be helpful to avoid printing the
'loadNamespace()' call on error (since it's mostly just scary /
uninformative), and check up-front that the package is installed
before attempting to call 'loadNamespace()'. I'm sure a number of
novice users will still just throw their hands up in the air and say
"I don't know what to do", but I think this would help steer a number
of users in the right direction.

(The prescription to suggest installing a package from CRAN if
available might be a step too far, but I think making it more clear
that the error is due to a missing dependent package would help.)

Any thoughts?
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] strange behavior in 'inherits' check for loaded S4 object

[Kevin Ushey]

Hi Martin, John,

Thanks for the responses! I've tidied up some of the notes from this
mailing list thread and posted them on the bug tracker.

John, in this case, I think namespaces are relevant because for
non-exported S4 classes, the class information is made available
through the '.__C__' symbol in the package's namespace, but
not the package environment that gets attached to the search path. In
this (rare, yet not impossible) sequence of events, it looks like R
attempts to resolve the '.__C__' symbol in the wrong
environment, and so class information lookup fails, and we end up
caching the wrong inheritance information.

Thanks,
Kevin

On Sun, Jul 31, 2016 at 5:12 AM, John Chambers  wrote:
> (Just returning from the "wilds" of Canada, so not able to comment on the 
> specifics, but ...)
>
> There is a basic point about generic functions that may be related to the 
> "private" class question and my earlier remarks that Martin alluded to.
>
> R (and S4 before it)  allows packages to define methods for a generic 
> function in another package.  Say, for plot() in graphics.
>
> The current model is that the generic plot() remains one function, 
> specifically a generic associated with the graphics package.
>
> Package A might define a method corresponding to one or two classes defined 
> in that package.  When A is loaded, those methods are added to the table for 
> plot() in the current session.
>
> Now suppose a user calls a function, foo(), in package B, and that foo() in 
> turn calls plot().  This is the same plot() function, and in particular will 
> include the methods supplied from package A.
>
> This is regardless of the two packages having any overt connection.  Also, 
> the methods are accepted by the generic function regardless of whether the 
> class is explicitly exported or not.  In this sense, classes cannot be 
> entirely private if methods are defined for a non-local function.  Namespaces 
> are not directly relevant.
>
> Whether this can lead to strange behavior isn't clear, and if so, is it a 
> sign that something undesirable was done in the particular example?  (In 
> Extending R, I suggested a "right to write methods" that  would discourage a 
> package from having methods unless it owned the function or some of the 
> classes.)
>
> R could adopt a different model for generic functions, where a package that 
> defined a method for a non-exported class would create a "local" version of 
> the generic, but that would likely raise some other issues.
>
> But seems like a useful topic for discussion.
>
> John
>
> On Jul 30, 2016, at 11:07 AM, Martin Maechler  
> wrote:
>
>>>>>>> Kevin Ushey 
>>>>>>>on Fri, 29 Jul 2016 11:46:19 -0700 writes:
>>
>>> I should add one more item that may be related here --
>>> calling 'methods:::.requirePackage' returns a different
>>> result based on whether the package namespace is already
>>> loaded or not.
>>
>>> If the package namespace is not loaded, the package is
>>> loaded and attached, and the package environment is
>>> returned:
>>
>>>> methods:::.requirePackage("digest")
>>>Loading required package: digest >> package:digest> attr(,"name") [1] "package:digest"
>>> attr(,"path") [1]
>>> "/Users/kevin/Library/R/3.3/library/digest"
>>>> "digest" %in% loadedNamespaces()
>>>[1] TRUE
>>>> "package:digest" %in% search()
>>>[1] TRUE
>>
>>> On the other hand, if the package namespace has already
>>> been loaded, the package namespace is returned without
>>> attaching the package:
>>
>>>> requireNamespace("digest")
>>>Loading required namespace: digest
>>>> methods:::.requirePackage("digest")
>>>
>>>> "digest" %in% loadedNamespaces()
>>>[1] TRUE
>>>> "package:digest" %in% search()
>>>[1] FALSE
>>
>>> This may be intentional, but the behavior seems surprising
>>> and could be responsible for the behavior outlined
>>> earlier.
>>
>> Yes, the behavior you outlined earlier is buggy, and I also have
>> seen similar bugous behavior for the case of non-exported
>> classes.
>>
>> Part of it is historical:  The S4 code was mostly written before
>> namespaces were introduced into R;   I vaguely remember John
>> Chambers (the principal creator of S4) saying that he did not
>> intend the formal classes

Re: [Rd] strange behavior in 'inherits' check for loaded S4 object

[Kevin Ushey]

I should add one more item that may be related here -- calling
'methods:::.requirePackage' returns a different result based on
whether the package namespace is already loaded or not.

If the package namespace is not loaded, the package is loaded and
attached, and the package environment is returned:

> methods:::.requirePackage("digest")
Loading required package: digest

attr(,"name")
[1] "package:digest"
attr(,"path")
[1] "/Users/kevin/Library/R/3.3/library/digest"
> "digest" %in% loadedNamespaces()
[1] TRUE
> "package:digest" %in% search()
[1] TRUE

On the other hand, if the package namespace has already been loaded,
the package namespace is returned without attaching the package:

> requireNamespace("digest")
Loading required namespace: digest
> methods:::.requirePackage("digest")

> "digest" %in% loadedNamespaces()
[1] TRUE
> "package:digest" %in% search()
[1] FALSE

This may be intentional, but the behavior seems surprising and could
be responsible for the behavior outlined earlier.

Best,
Kevin

On Fri, Jul 29, 2016 at 11:37 AM, Kevin Ushey  wrote:
> I have a small idea as to what's going on now; at least, why exporting
> the class resolves this particular issue.
>
> Firstly, when an S4 class is not exported, the associated
> '.__C__' object is not made part of the package environment.
> For example, I see:
>
> > getAnywhere(".__C__SubMatrix")
> A single object matching '.__C__SubMatrix' was found
> It was found in the following places
>   namespace:s4inherits
> with value
> < ... >
>
> Note that the symbol is only discovered in the package namespace. When
> the class is exported (e.g. with 'exportClasses(SubMatrix)' in the
> NAMESPACE file), it's found both in 'package:s4inherits' and
> 'namespace:s4inherits'.
>
> Secondly, when R attempts to resolve the superclasses for an S3 class,
> the function 'methods:::.extendsForS3' is called. Tracing that code
> eventually gets us here:
>
> https://github.com/wch/r-source/blob/trunk/src/library/methods/R/SClasses.R#L255
>
> Note that we reach this code path as the S3 class cache has not been
> populated yet; ie, this code returns NULL:
>
> https://github.com/wch/r-source/blob/trunk/src/library/methods/R/SClasses.R#L238-L240
>
> So, the class hierarchy is looked up using this code:
>
> if(isTRUE(nzchar(package))) {
>whereP <- .requirePackage(package)
>value <- get0(cname, whereP, inherits = inherits)
> }
>
> However, because the '.__C__SubMatrix' object is only made available
> in the package's namespace, not within the package environment, it is
> not resolved, and so lookup fails. (Presumedly, that lookup is done
> when initially building a cache for S3 dispatch?) So, I wonder if that
> class lookup should occur within the package's namespace instead?
>
> Thanks for your time,
> Kevin
>
> On Sat, Jun 25, 2016 at 12:46 PM, Kevin Ushey  wrote:
>> Hi,
>>
>> (sorry for the wall of text; the issue here appears to be rather complicated)
>>
>> I'm seeing a somewhat strange case where checking whether an S4 object
>> inherits from a parent class defined from another package with
>> 'inherits' fails if that object is materialized through a call to
>> 'load'. That's a mouthful, so I've put together a relatively small
>> reproducible example online here:
>>
>> https://github.com/kevinushey/s4inherits
>>
>> This package, 's4inherits', defines an S4 class, 'SubMatrix', that
>> inherits from the 'dsyMatrix' class defined in the Matrix package.
>> After installing the package, I run some simple tests:
>>
>> $ R -f test-save.R
>>
>>> library(s4inherits)
>>> data <- SubMatrix(1)
>>>
>>> is(data, "SubMatrix")
>> [1] TRUE
>>> inherits(data, "SubMatrix")
>> [1] TRUE
>>>
>>> is(data, "dsyMatrix")
>> [1] TRUE
>>> inherits(data, "dsyMatrix")
>> [1] TRUE
>>>
>>> save(data, file = "test.RData")
>>>
>>
>> All the inheritance checks report as we would expect. I check that the
>> inheritance reports are as expected, then save that object to
>> 'test.RData'. I then load that data file in a new R session and run
>> the same checks:
>>
>> $ R -f test-load.R
>>
>>> li

Re: [Rd] strange behavior in 'inherits' check for loaded S4 object

[Kevin Ushey]

I have a small idea as to what's going on now; at least, why exporting
the class resolves this particular issue.

Firstly, when an S4 class is not exported, the associated
'.__C__' object is not made part of the package environment.
For example, I see:

> getAnywhere(".__C__SubMatrix")
A single object matching '.__C__SubMatrix' was found
It was found in the following places
  namespace:s4inherits
with value
< ... >

Note that the symbol is only discovered in the package namespace. When
the class is exported (e.g. with 'exportClasses(SubMatrix)' in the
NAMESPACE file), it's found both in 'package:s4inherits' and
'namespace:s4inherits'.

Secondly, when R attempts to resolve the superclasses for an S3 class,
the function 'methods:::.extendsForS3' is called. Tracing that code
eventually gets us here:

https://github.com/wch/r-source/blob/trunk/src/library/methods/R/SClasses.R#L255

Note that we reach this code path as the S3 class cache has not been
populated yet; ie, this code returns NULL:

https://github.com/wch/r-source/blob/trunk/src/library/methods/R/SClasses.R#L238-L240

So, the class hierarchy is looked up using this code:

if(isTRUE(nzchar(package))) {
   whereP <- .requirePackage(package)
   value <- get0(cname, whereP, inherits = inherits)
}

However, because the '.__C__SubMatrix' object is only made available
in the package's namespace, not within the package environment, it is
not resolved, and so lookup fails. (Presumedly, that lookup is done
when initially building a cache for S3 dispatch?) So, I wonder if that
class lookup should occur within the package's namespace instead?

Thanks for your time,
Kevin

On Sat, Jun 25, 2016 at 12:46 PM, Kevin Ushey  wrote:
> Hi,
>
> (sorry for the wall of text; the issue here appears to be rather complicated)
>
> I'm seeing a somewhat strange case where checking whether an S4 object
> inherits from a parent class defined from another package with
> 'inherits' fails if that object is materialized through a call to
> 'load'. That's a mouthful, so I've put together a relatively small
> reproducible example online here:
>
> https://github.com/kevinushey/s4inherits
>
> This package, 's4inherits', defines an S4 class, 'SubMatrix', that
> inherits from the 'dsyMatrix' class defined in the Matrix package.
> After installing the package, I run some simple tests:
>
> $ R -f test-save.R
>
>> library(s4inherits)
>> data <- SubMatrix(1)
>>
>> is(data, "SubMatrix")
> [1] TRUE
>> inherits(data, "SubMatrix")
> [1] TRUE
>>
>> is(data, "dsyMatrix")
> [1] TRUE
>> inherits(data, "dsyMatrix")
> [1] TRUE
>>
>> save(data, file = "test.RData")
>>
>
> All the inheritance checks report as we would expect. I check that the
> inheritance reports are as expected, then save that object to
> 'test.RData'. I then load that data file in a new R session and run
> the same checks:
>
> $ R -f test-load.R
>
>> library(methods)
>> load("test.RData")
>>
>> inherits(data, "SubMatrix")
> Loading required package: s4inherits
> [1] TRUE
>> is(data, "SubMatrix")
> [1] TRUE
>>
>> inherits(data, "dsyMatrix")
> [1] FALSE # (??)
>> is(data, "dsyMatrix")
> [1] TRUE
>>
>
> Note that R now reports that my loaded object does _not_ inherit from
> "dsyMatrix", yet this occurs only when checked with 'inherits()' --
> 'is' produces the expected result.
>
> I do not see the behavior if I explicitly load / attach the
> 's4inherits' package before loading the associated data file; it only
> occurs if the package namespace is loaded in response to loading the
> data object hosting a 'SubMatrix' object.
>
> More precisely, the package namespace is loaded when the promise
> hosting the data object is evaluated; that promise being generated by
> 'load', if I understand correctly. Somehow, evaluation of that promise
> within the call to 'inherits' in this very special case causes the
> unexpected behavior -- ie, if the first thing that you do with the
> loaded object is check its class with 'inherits', then you get this
> unexpected result.
>
> Even more, this behavior seems to go away if the 's4inherits' package
> explicitly exports the class -- however, you could imagine this class
> normally being internal to the package, and so it may be undesirable
> to export it.
>
> I checked a bit into the C code, and IIU

Re: [Rd] package installation fails when symlink of same name exists

[Kevin Ushey]

On Wed, Jul 20, 2016 at 3:52 AM, Martin Maechler
 wrote:
>>>>>> Jeroen Ooms 
>>>>>> on Wed, 20 Jul 2016 10:26:19 +0200 writes:
>
>     > On Tue, Jul 19, 2016 at 6:46 PM, Kevin Ushey  
> wrote:
> >> R fails to install a package from source over a pre-existing package
> >> when the path to that package is a symlink, rather than a directory.
> >> ...
> >> I don't think anyone's reported this being an issue before
>
> > I ran into this as well a while back:
> >   https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16725
>
> I've now at least "acknowledged" that bug report.
> and have looked into changing the  is_subdir() function so it
> returns TRUE in the case of a symlink [on those platforms where
> Sys.readlink() "works", i.e., supposedly not on Windows; however
> that maybe sufficient to close that bug report and also Kevin's
> issue, right ?]
>
> However, Kevin, in his posting, continues
>
> > I guess my wish here would be that R would check if any file already
> > existed at the 'instdir' path, and if it existed and was a symlink, R
> > would remove that symlink before install.
>
> are you sure?
> I think ... and from what you mention below ("packrat") it would
> rather be important to *keep* the symlink, and install to
> whereever the symlink is pointing,  no ?

For packrat's case at least, removing the symlink and installing to a
newly-created directory within the library would be fine -- later,
when a user wants to 'save the state' of their library, they would
call 'packrat::snapshot()', and that call would take care of moving
the newly-installed package to the cache and restoring the symlink as
required.

That said, installing within the symlinked directory would definitely
be nice :-) I just thought the request might be out of scope.

> > It could happen before creating the directory, e.g. here:
>
> > 
> https://github.com/wch/r-source/blob/62f5acbdbdf36e1fc618510312125d1677d79941/src/library/tools/R/install.R#L277-L281
>
> > One thing that was a bit surprising to me -- R does not remove a
> > pre-existing package installation if it exists (when installing from
> > source), it merely installs over it, so files / artifacts from a
> > previous package installation could be left over after installing a
> > new package. It seems this is not a problem in practice since I don't
> > think anyone's reported this being an issue before, but for hygiene it
> > seems like a pre-existing directory could / should be removed when
> > installing a new package. (It appears that R does clear out a
> > pre-existing directory when downloading and installing a package
> > binary directly from CRAN.)
>
> Well, at least with  update.packages()  it seems natural to me
> that R would not just remove all previous parts there ..
>
> > For motivation: I bumped into this when attempting to implement a
> > package caching feature with packrat. A packrat project using a global
> > cache will have a (private) R library containing symlinks to R package
> > installations in a separate, global library. This allows projects to
> > effectively be isolated from one another, while avoiding duplication
> > of packages used across multiple projects.
>
> Yes, I found this a nice feature when I heard about packrat.
>
> But then, really R should *not* remove the symlink and create a
> regular subdirectory in that library there !

I agree this would be ideal, I just thought this request might be out
of scope, since the typical use case for R libraries is a
directory-of-directories, not a directory-of-symlinks-to-directories
(although packrat has had a lot of success with the second scenario!)

Thanks, Martin!

> > Unfortunately, some packrat
> > users bump into this when attempting to update a package that has
> > entered the cache (and so is a symlink in their R library).
>
> > Thanks for your time,
> > Kevin
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] package installation fails when symlink of same name exists

[Kevin Ushey]

R fails to install a package from source over a pre-existing package
when the path to that package is a symlink, rather than a directory. A
reproducible example to illustrate (using MASS as an example):

# create a temporary R library in tempdir
library <- tempfile()
if (!dir.exists(library)) dir.create(library)

# symlink MASS from system library to temporary library
MASS <- find.package("MASS", lib.loc = .Library)
file.symlink(MASS, file.path(library, "MASS"))

# use our custom library, and try to install MASS
.libPaths(library)
utils::install.packages("MASS", lib = library, type = "source")

On running this, I see a surprising error message:

ERROR: 'MASS' is not a legal package name

The code that I'm bumping into lives here:

https://github.com/wch/r-source/blob/62f5acbdbdf36e1fc618510312125d1677d79941/src/library/tools/R/install.R#L283-L287

I guess my wish here would be that R would check if any file already
existed at the 'instdir' path, and if it existed and was a symlink, R
would remove that symlink before install. It could happen before
creating the directory, e.g. here:

https://github.com/wch/r-source/blob/62f5acbdbdf36e1fc618510312125d1677d79941/src/library/tools/R/install.R#L277-L281

One thing that was a bit surprising to me -- R does not remove a
pre-existing package installation if it exists (when installing from
source), it merely installs over it, so files / artifacts from a
previous package installation could be left over after installing a
new package. It seems this is not a problem in practice since I don't
think anyone's reported this being an issue before, but for hygiene it
seems like a pre-existing directory could / should be removed when
installing a new package. (It appears that R does clear out a
pre-existing directory when downloading and installing a package
binary directly from CRAN.)

For motivation: I bumped into this when attempting to implement a
package caching feature with packrat. A packrat project using a global
cache will have a (private) R library containing symlinks to R package
installations in a separate, global library. This allows projects to
effectively be isolated from one another, while avoiding duplication
of packages used across multiple projects. Unfortunately, some packrat
users bump into this when attempting to update a package that has
entered the cache (and so is a symlink in their R library).

Thanks for your time,
Kevin

---

$ R --vanilla --slave -e 'sessionInfo()' | head -n 3
R version 3.3.0 Patched (2016-05-23 r70665)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.5 (El Capitan)

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] strange behavior in 'inherits' check for loaded S4 object

[Kevin Ushey]

Hi,

(sorry for the wall of text; the issue here appears to be rather complicated)

I'm seeing a somewhat strange case where checking whether an S4 object
inherits from a parent class defined from another package with
'inherits' fails if that object is materialized through a call to
'load'. That's a mouthful, so I've put together a relatively small
reproducible example online here:

https://github.com/kevinushey/s4inherits

This package, 's4inherits', defines an S4 class, 'SubMatrix', that
inherits from the 'dsyMatrix' class defined in the Matrix package.
After installing the package, I run some simple tests:

$ R -f test-save.R

> library(s4inherits)
> data <- SubMatrix(1)
>
> is(data, "SubMatrix")
[1] TRUE
> inherits(data, "SubMatrix")
[1] TRUE
>
> is(data, "dsyMatrix")
[1] TRUE
> inherits(data, "dsyMatrix")
[1] TRUE
>
> save(data, file = "test.RData")
>

All the inheritance checks report as we would expect. I check that the
inheritance reports are as expected, then save that object to
'test.RData'. I then load that data file in a new R session and run
the same checks:

$ R -f test-load.R

> library(methods)
> load("test.RData")
>
> inherits(data, "SubMatrix")
Loading required package: s4inherits
[1] TRUE
> is(data, "SubMatrix")
[1] TRUE
>
> inherits(data, "dsyMatrix")
[1] FALSE # (??)
> is(data, "dsyMatrix")
[1] TRUE
>

Note that R now reports that my loaded object does _not_ inherit from
"dsyMatrix", yet this occurs only when checked with 'inherits()' --
'is' produces the expected result.

I do not see the behavior if I explicitly load / attach the
's4inherits' package before loading the associated data file; it only
occurs if the package namespace is loaded in response to loading the
data object hosting a 'SubMatrix' object.

More precisely, the package namespace is loaded when the promise
hosting the data object is evaluated; that promise being generated by
'load', if I understand correctly. Somehow, evaluation of that promise
within the call to 'inherits' in this very special case causes the
unexpected behavior -- ie, if the first thing that you do with the
loaded object is check its class with 'inherits', then you get this
unexpected result.

Even more, this behavior seems to go away if the 's4inherits' package
explicitly exports the class -- however, you could imagine this class
normally being internal to the package, and so it may be undesirable
to export it.

I checked a bit into the C code, and IIUC the check here looks up the
class hierarchy in the R_S4_extends_table object defined in
'attrib.c', so it seems like that mapping is potentially not getting
constructed with the full hierarchy (although hopefully someone with
more knowledge of the S4 internals + interaction with S3 can
elaborate).

(FWIW, this was distilled from a case where S3 dispatch on a similar
loaded S4 object failed, due to failure to resolve the class hierarchy
for the S3 dispatch context.)

Thanks,
Kevin

---

$ R --slave -e "utils::sessionInfo()"
R Under development (unstable) (2016-06-13 r70769)
Platform: x86_64-apple-darwin15.5.0 (64-bit)
Running under: OS X 10.11.5 (El Capitan)

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Kevin Ushey]

Thanks for looking into this so promptly!

Should users expect the behaviour to be congruent across all of the
supported external programs (curl, wget) as well? E.g.

URL <- "http://cran.rstudio.org/no/such/file/here.tar.gz";
download <- function(file, method, ...)
  print(download.file(file, destfile = tempfile(), method = method, ...))

download(URL, method = "internal") ## error
download(URL, method = "curl") ## status code 0
download(URL, method = "wget") ## warning (status code 8)
download(URL, method = "libcurl") ## status code 0

It seems unfortunate that the behaviour differs across each method; at
least in my mind `download.file()` should be a unified interface that
tries to do the 'same thing' regardless of the chosen method.

FWIW, one can force 'curl' to fail on HTTP error codes (-f) and this
can be passed down by R, e.g.

download(URL, method = "curl", extra = "-f") ## warning (status code 22)

but I still think this should be promoted to an error rather than a
warning. (Of course, changing that would imply a backwards
incompatible change; however, I think it would be the correct change).

(PS: I just tested r69197 and method = "libcurl" does indeed report an
error now in the above test case on my system [OS X]; thanks!)

Kevin


On Thu, Aug 27, 2015 at 10:27 AM, Martin Morgan  wrote:
> R-devel r69197 returns appropriate errors for the cases below; I know of a
> few rough edges
>
> - ftp error codes are not reported correctly
> - download.file creates destfile before discovering that http fails, leaving
> an empty file on disk
>
> and am happy to hear of more.
>
> Martin
>
>
> On 08/27/2015 08:46 AM, Jeroen Ooms wrote:
>>
>> On Thu, Aug 27, 2015 at 5:16 PM, Martin Maechler
>>  wrote:
>>>
>>> Probably I'm confused now...
>>> Both R-patched and R-devel give an error (after a *long* wait!)
>>> for
>>> download.file("https://someserver.com/mydata.csv";, "mydata.csv")
>>>
>>> So that problem is I think  solved now.
>>
>>
>> I'm sorry for the confusion, this was a hypothetical example.
>> Connection failures are different from http status errors. Below some
>> real examples of servers returning http errors. For each example the
>> "internal" method correctly raises an R error, whereas the "libcurl"
>> method does not.
>>
>> # File not found (404)
>> download.file("http://httpbin.org/data.csv";, "data.csv", method =
>> "internal")
>> download.file("http://httpbin.org/data.csv";, "data.csv", method =
>> "libcurl")
>> readLines(url("http://httpbin.org/data.csv";, method = "internal"))
>> readLines(url("http://httpbin.org/data.csv";, method = "libcurl"))
>>
>> # Unauthorized (401)
>> download.file("https://httpbin.org/basic-auth/user/passwd";,
>> "data.csv", method = "internal")
>> download.file("https://httpbin.org/basic-auth/user/passwd";,
>> "data.csv", method = "libcurl")
>> readLines(url("https://httpbin.org/basic-auth/user/passwd";, method =
>> "internal"))
>> readLines(url("https://httpbin.org/basic-auth/user/passwd";, method =
>> "libcurl"))
>>
>
>
> --
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M1 B861
> Phone: (206) 667-2793
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Kevin Ushey]

(final post, sorry to be spamming everyone all day...)

As kindly pointed out by Martin off-list, I was in fact using an old
version of R-devel (it looks like the binaries provided at
http://r.research.att.com/ are currently stale -- although the page
lists r69167 as the current version, the binaries being distributed
are for r69078).

Building R locally with trunk (r69180) and testing confirms that
errors no longer clobber the whole `install.packages()` process;
having the various download methods respect HTTP status / error codes
when using `libcurl` is still an issue but one I imagine that R-core
is aware of.

Thanks, and apologies again for the spam,
Kevin

On Tue, Aug 25, 2015 at 2:41 PM, Kevin Ushey  wrote:
> In fact, this does reproduce on R-devel:
>
> > options(download.file.method = "libcurl")
> > options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra =
> + "http://www.stats.ox.ac.uk/pub/RWin";))
> > install.packages("lattice") ## could be any package
> Installing package into ‘/Users/kevinushey/Library/R/3.3/library’
> (as ‘lib’ is unspecified)
> Error: Line starting '
> > sessionInfo()
> R Under development (unstable) (2015-08-14 r69078)
> Platform: x86_64-apple-darwin13.4.0 (64-bit)
> Running under: OS X 10.10.4 (Yosemite)
>
> I think this could be problematic for users with custom CRAN
> repositories. For example, if I have a CRAN repository that only
> serves source packages (no binary packages), this implies that any R
> session configured to download binary packages would fail to download
> any packages at all (as it would barf on attempting to read the
> non-existent PACKAGES file for the 'binary' branch of the custom
> repository).
>
> This can also be seen by attempting to install a package using current
> R-devel (since no binaries are made available for R 3.3):
>
> > options(download.file.method = "libcurl")
> > options(repos = c(CRAN = "https://cran.rstudio.com/";))
> > print(getOption("pkgType"))
> [1] "both"
> > install.packages("lattice")
> Installing package into ‘/Users/kevinushey/Library/R/3.3/library’
> (as ‘lib’ is unspecified)
> Error in install.packages : Line starting ' ...' is malformed!
>
> The same error (with a different, XML response) is returned when using
> e.g. `https://cran.fhcrc.org`.
>
> Kevin
>
> On Tue, Aug 25, 2015 at 1:33 PM, Martin Morgan  wrote:
>> On 08/25/2015 01:30 PM, Kevin Ushey wrote:
>>>
>>> Hi Martin,
>>>
>>> Indeed it does (and I should have confirmed myself with R-patched and
>>> R-devel
>>> before posting...)
>>
>>
>> actually I don't know that it does -- it addresses the symptom but I think
>> there should be an error from libcurl on the 403 / 404 rather than from
>> read.dcf on error page...
>>
>> Martin
>>
>>
>>>
>>> Thanks, and sorry for the noise.
>>> Kevin
>>>
>>>
>>> On Tue, Aug 25, 2015, 13:11 Martin Morgan >> <mailto:mtmor...@fredhutch.org>> wrote:
>>>
>>> On 08/25/2015 12:54 PM, Kevin Ushey wrote:
>>>  > Hi all,
>>>  >
>>>  > The following fails for me (on OS X, although I imagine it's the
>>> same
>>>  > on other platforms using libcurl):
>>>  >
>>>  >  options(download.file.method = "libcurl")
>>>  >  options(repos = c(CRAN = "https://cran.rstudio.com/";,
>>> CRANextra =
>>>  > "http://www.stats.ox.ac.uk/pub/RWin";))
>>>  >  install.packages("lattice") ## could be any package
>>>  >
>>>  > gives me:
>>>  >
>>>  >  > options(download.file.method = "libcurl")
>>>  >  > options(repos = c(CRAN = "https://cran.rstudio.com/";,
>>> CRANextra
>>>  > = "http://www.stats.ox.ac.uk/pub/RWin";))
>>>  >  > install.packages("lattice") ## coudl be any package
>>>  >  Installing package into
>>> ‘/Users/kevinushey/Library/R/3.2/library’
>>>  >  (as ‘lib’ is unspecified)
>>>  >  Error: Line starting '>>  >
>>>  > This seems to come from a call to `available.packages()` to a URL
>>> that
>>>  > doesn't exist on the server (likely when querying PACKAGES on the
>>>  &g

Re: [Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Kevin Ushey]

In fact, this does reproduce on R-devel:

> options(download.file.method = "libcurl")
> options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra =
+ "http://www.stats.ox.ac.uk/pub/RWin";))
> install.packages("lattice") ## could be any package
Installing package into ‘/Users/kevinushey/Library/R/3.3/library’
(as ‘lib’ is unspecified)
Error: Line starting ' sessionInfo()
R Under development (unstable) (2015-08-14 r69078)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.4 (Yosemite)

I think this could be problematic for users with custom CRAN
repositories. For example, if I have a CRAN repository that only
serves source packages (no binary packages), this implies that any R
session configured to download binary packages would fail to download
any packages at all (as it would barf on attempting to read the
non-existent PACKAGES file for the 'binary' branch of the custom
repository).

This can also be seen by attempting to install a package using current
R-devel (since no binaries are made available for R 3.3):

> options(download.file.method = "libcurl")
> options(repos = c(CRAN = "https://cran.rstudio.com/";))
> print(getOption("pkgType"))
[1] "both"
> install.packages("lattice")
Installing package into ‘/Users/kevinushey/Library/R/3.3/library’
(as ‘lib’ is unspecified)
Error in install.packages : Line starting 'https://cran.fhcrc.org`.

Kevin

On Tue, Aug 25, 2015 at 1:33 PM, Martin Morgan  wrote:
> On 08/25/2015 01:30 PM, Kevin Ushey wrote:
>>
>> Hi Martin,
>>
>> Indeed it does (and I should have confirmed myself with R-patched and
>> R-devel
>> before posting...)
>
>
> actually I don't know that it does -- it addresses the symptom but I think
> there should be an error from libcurl on the 403 / 404 rather than from
> read.dcf on error page...
>
> Martin
>
>
>>
>> Thanks, and sorry for the noise.
>> Kevin
>>
>>
>> On Tue, Aug 25, 2015, 13:11 Martin Morgan > <mailto:mtmor...@fredhutch.org>> wrote:
>>
>> On 08/25/2015 12:54 PM, Kevin Ushey wrote:
>>  > Hi all,
>>  >
>>  > The following fails for me (on OS X, although I imagine it's the
>> same
>>  > on other platforms using libcurl):
>>  >
>>  >  options(download.file.method = "libcurl")
>>  >  options(repos = c(CRAN = "https://cran.rstudio.com/";,
>> CRANextra =
>>  > "http://www.stats.ox.ac.uk/pub/RWin";))
>>  >  install.packages("lattice") ## could be any package
>>  >
>>  > gives me:
>>  >
>>  >  > options(download.file.method = "libcurl")
>>  >  > options(repos = c(CRAN = "https://cran.rstudio.com/";,
>> CRANextra
>>  > = "http://www.stats.ox.ac.uk/pub/RWin";))
>>  >  > install.packages("lattice") ## coudl be any package
>>  >  Installing package into
>> ‘/Users/kevinushey/Library/R/3.2/library’
>>  >  (as ‘lib’ is unspecified)
>>  >  Error: Line starting '>  >
>>  > This seems to come from a call to `available.packages()` to a URL
>> that
>>  > doesn't exist on the server (likely when querying PACKAGES on the
>>  > CRANextra repo)
>>  >
>>  > Eg.
>>  >
>>  >  > URL <- "http://www.stats.ox.ac.uk/pub/RWin";
>>  >  > available.packages(URL, method = "internal")
>>  >  Warning: unable to access index for repository
>>  > http://www.stats.ox.ac.uk/pub/RWin
>>  >   Package Version Priority Depends Imports LinkingTo
>> Suggests
>>  > Enhances License License_is_FOSS
>>  >  License_restricts_use OS_type Archs MD5sum
>> NeedsCompilation
>>  > File Repository
>>  >  > available.packages(URL, method = "libcurl")
>>  >  Error: Line starting '>  >
>>  > It looks like libcurl downloads and retrieves the 403 page itself,
>>  > rather than reporting that it was actually forbidden, e.g.:
>>  >
>>  >  >
>>
>> download.file("http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz";,
>>  > tempfile(), method = "libcurl")
>>  > 

Re: [Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Kevin Ushey]

Hi Martin,

Indeed it does (and I should have confirmed myself with R-patched and
R-devel before posting...)

Thanks, and sorry for the noise.
Kevin

On Tue, Aug 25, 2015, 13:11 Martin Morgan  wrote:

> On 08/25/2015 12:54 PM, Kevin Ushey wrote:
> > Hi all,
> >
> > The following fails for me (on OS X, although I imagine it's the same
> > on other platforms using libcurl):
> >
> >  options(download.file.method = "libcurl")
> >  options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra =
> > "http://www.stats.ox.ac.uk/pub/RWin";))
> >  install.packages("lattice") ## could be any package
> >
> > gives me:
> >
> >  > options(download.file.method = "libcurl")
> >  > options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra
> > = "http://www.stats.ox.ac.uk/pub/RWin";))
> >  > install.packages("lattice") ## coudl be any package
> >  Installing package into ‘/Users/kevinushey/Library/R/3.2/library’
> >  (as ‘lib’ is unspecified)
> >  Error: Line starting ' >
> > This seems to come from a call to `available.packages()` to a URL that
> > doesn't exist on the server (likely when querying PACKAGES on the
> > CRANextra repo)
> >
> > Eg.
> >
> >  > URL <- "http://www.stats.ox.ac.uk/pub/RWin";
> >  > available.packages(URL, method = "internal")
> >  Warning: unable to access index for repository
> > http://www.stats.ox.ac.uk/pub/RWin
> >   Package Version Priority Depends Imports LinkingTo Suggests
> > Enhances License License_is_FOSS
> >  License_restricts_use OS_type Archs MD5sum NeedsCompilation
> > File Repository
> >  > available.packages(URL, method = "libcurl")
> >  Error: Line starting ' >
> > It looks like libcurl downloads and retrieves the 403 page itself,
> > rather than reporting that it was actually forbidden, e.g.:
> >
> >  > download.file("
> http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz
> ",
> > tempfile(), method = "libcurl")
> >  trying URL '
> http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz
> '
> >  Content type 'text/html; charset=iso-8859-1' length 339 bytes
> >  ==
> >  downloaded 339 bytes
> >
> > Using `method = "internal"` gives an error related to the inability to
> > access that URL due to the HTTP status 403.
> >
> > The overarching issue here is that package installation shouldn't fail
> > even if libcurl fails to access one of the repositories set.
> >
>
> With
>
>  > R.version.string
> [1] "R version 3.2.2 Patched (2015-08-25 r69179)"
>
> the behavior is to warn with an indication of the repository for which the
> problem occurs
>
>  > URL <- "http://www.stats.ox.ac.uk/pub/RWin";
>  > available.packages(URL, method="libcurl")
> Warning: unable to access index for repository
> http://www.stats.ox.ac.uk/pub/RWin:
>Line starting '   Package Version Priority Depends Imports LinkingTo Suggests Enhances
>   License License_is_FOSS License_restricts_use OS_type Archs MD5sum
>   NeedsCompilation File Repository
>  > available.packages(URL, method="internal")
> Warning: unable to access index for repository
> http://www.stats.ox.ac.uk/pub/RWin:
>cannot open URL 'http://www.stats.ox.ac.uk/pub/RWin/PACKAGES'
>   Package Version Priority Depends Imports LinkingTo Suggests Enhances
>   License License_is_FOSS License_restricts_use OS_type Archs MD5sum
>   NeedsCompilation File Repository
>
> Does that work for you / address the problem?
>
> Martin
>
> >> sessionInfo()
> > R version 3.2.2 (2015-08-14)
> > Platform: x86_64-apple-darwin13.4.0 (64-bit)
> > Running under: OS X 10.10.4 (Yosemite)
> >
> > locale:
> > [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
> >
> > attached base packages:
> > [1] stats graphics  grDevices utils datasets  methods   base
> >
> > other attached packages:
> > [1] testthat_0.8.1.0.99  knitr_1.11   devtools_1.5.0.9001
> > [4] BiocInstaller_1.15.5
> >
> > loaded via a namespace (and not attached):
> >   [1] httr_1.0.0 R6_2.0.0.9000  tools_3.2.2parallel_3.2.2
> whisker_0.3-2
> >   [6] RCurl_1.95-4.1 memoise_0.2.1  stringr_0.6.2  digest_0.6.4
>  evaluate_0.7.2
> >
> > Thanks,
> > Kevin
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>
>
> --
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M1 B861
> Phone: (206) 667-2793
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Kevin Ushey]

Hi all,

The following fails for me (on OS X, although I imagine it's the same
on other platforms using libcurl):

options(download.file.method = "libcurl")
options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra =
"http://www.stats.ox.ac.uk/pub/RWin";))
install.packages("lattice") ## could be any package

gives me:

> options(download.file.method = "libcurl")
> options(repos = c(CRAN = "https://cran.rstudio.com/";, CRANextra
= "http://www.stats.ox.ac.uk/pub/RWin";))
> install.packages("lattice") ## coudl be any package
Installing package into ‘/Users/kevinushey/Library/R/3.2/library’
(as ‘lib’ is unspecified)
Error: Line starting ' URL <- "http://www.stats.ox.ac.uk/pub/RWin";
> available.packages(URL, method = "internal")
Warning: unable to access index for repository
http://www.stats.ox.ac.uk/pub/RWin
 Package Version Priority Depends Imports LinkingTo Suggests
Enhances License License_is_FOSS
License_restricts_use OS_type Archs MD5sum NeedsCompilation
File Repository
> available.packages(URL, method = "libcurl")
Error: Line starting ' 
download.file("http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz";,
tempfile(), method = "libcurl")
trying URL 
'http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz'
Content type 'text/html; charset=iso-8859-1' length 339 bytes
==
downloaded 339 bytes

Using `method = "internal"` gives an error related to the inability to
access that URL due to the HTTP status 403.

The overarching issue here is that package installation shouldn't fail
even if libcurl fails to access one of the repositories set.

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.4 (Yosemite)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

other attached packages:
[1] testthat_0.8.1.0.99  knitr_1.11   devtools_1.5.0.9001
[4] BiocInstaller_1.15.5

loaded via a namespace (and not attached):
 [1] httr_1.0.0 R6_2.0.0.9000  tools_3.2.2parallel_3.2.2 whisker_0.3-2
 [6] RCurl_1.95-4.1 memoise_0.2.1  stringr_0.6.2  digest_0.6.4   evaluate_0.7.2

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R package with Fortran module on Windows? undefined reference to `__stack_chk_fail'

[Kevin Ushey]

You should be able to set PKG_FCFLAGS="-fno-stack-protector" when
compiling to ensure that the stack protector is not used.

(Trying that out on a Windows VM, with a simple `R CMD build` + `R CMD
INSTALL`, compilation of your package succeeded but linking failed
saying the DLL 'Fpi' was not found; I imagine that's a separate
issue.)

As an aside, be sure to check out what R-exts has to say on the topic
of Fortran code in R packages (especially F90 and above):
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-F95-code.

Cheers,
Kevin

On Mon, Jul 27, 2015 at 11:06 AM, Ignacio Martinez  wrote:
> Hi,
>
> I created a R library that uses a Fortran module. Everything works like a
> charm on linux.
>
> Now I'm trying to make it work on Windows. I cloned my git repository
>  on a windows computer, and when I press
> the build and reload button on Rstudio I get these errors:
>
> ==> Rcmd.exe INSTALL --no-multiarch --with-keep.source MyPi
> * installing to library
> 'C:/Users/IMartinez/Documents/R/R-3.2.1/library'* installing *source*
> package 'MyPi' ...** libs
> gfortran -m64 -shared -s -static-libgcc -o MyPi.dll tmp.def Fpi.o
> -Ld:/RCompile/r-compiling/local/local320/lib/x64
> -Ld:/RCompile/r-compiling/local/local320/lib
> -LC:/Users/IMARTI~1/DOCUME~1/R/R-32~1.1/bin/x64 -lR
> Fpi.o: In function `__fortranpi_MOD_dboard':
> Fpi.f90:(.text+0xd7): undefined reference to `__stack_chk_fail'
> Fpi.o: In function `pi_':
> Fpi.f90:(.text+0x249): undefined reference to `__stack_chk_fail'
> collect2: ld returned 1 exit status
> no DLL was created
> ERROR: compilation failed for package 'MyPi'* removing
> 'C:/Users/IMartinez/Documents/R/R-3.2.1/library/MyPi'
>
> Exited with status 1.
>
>
> This is  the Fortran code:
>
>
> Module Fortranpi
> IMPLICIT NONE
> contains
> subroutine dboard(darts, dartsscore)
>   integer, intent(in):: darts
>   double precision, intent(out)  :: dartsscore
>   double precision   :: x_coord, y_coord
>   integer:: score, n
>
> score = 0
> do n = 1, darts
>   call random_number(x_coord)
>   call random_number(y_coord)
>
>   if ((x_coord**2 + y_coord**2) <= 1.0d0) then
>   score = score + 1
>   end if
> end do
>
> dartsscore = 4.0d0*score/darts
>
> end subroutine dboard
>
> subroutine pi(avepi, DARTS, ROUNDS) bind(C, name="pi_")
>   use, intrinsic :: iso_c_binding, only :
> c_double, c_int
>   real(c_double), intent(out)::  avepi
>   integer(c_int), intent(in) ::  DARTS, ROUNDS
>   integer::  MASTER, rank, i, n
>   integer, allocatable   ::  seed(:)
>   double precision   ::  pi_est, homepi, pirecv, pisum
> ! we set it to zero in the sequential run
> rank = 0! initialize the random number generator! we make sure the
> seed is different for each task
> call random_seed()
> call random_seed(size = n)
> allocate(seed(n))
> seed = 12 + rank*11
> call random_seed(put=seed(1:n))
> deallocate(seed)
>
> avepi = 0
> do i = 0, ROUNDS-1
>   call dboard(darts, pi_est)
>   ! calculate the average value of pi over all iterations
>   avepi = ((avepi*i) + pi_est)/(i + 1)
> end do
> end subroutine pi
>
> end module Fortranpi
>
>
> I tried adding  -fno-stack-protector
> -lssp but it did not help.
>
> I also tried doing this "by hand"  and
> I get these errors:
>
>
>> system("R CMD SHLIB -fno-stack-protector -lssp ./src/Fpi.f90")
> gfortran -m64 -shared -s -static-libgcc -o src/Fpi.dll tmp.def
> ./src/Fpi.o -fno-stack-protector -lssp
> -Ld:/RCompile/r-compiling/local/local320/lib/x64
> -Ld:/RCompile/r-compiling/local/local320/lib
> -LC:/Users/IMARTI~1/DOCUME~1/R/R-32~1.1/bin/x64 -lR>
> dyn.load("./src/Fpi.dll")
> Error in inDL(x, as.logical(local), as.logical(now), ...) :
>   unable to load shared object 
> 'C:/Users/IMartinez/Projects/MyPi/./src/Fpi.dll':
>   LoadLibrary failure:  %1 is not a valid Win32 application.
> 'C:/Users/IMartinez/Projects/MyPi/./src/Fpi.dll':
>   LoadLibrary failure:  %1 is not a valid Win32 application.
>
>
> Thanks for the help!
> …
>
>
> PS: I posted this question in stackoverflow with no luck.
> 
>
>
> PPS: I also sent this to r-help but they told me to try this list
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] bquote/evalq behavior changed in R-3.2.1

[Kevin Ushey]

My best guess is that it could be related to this commit:
https://github.com/wch/r-source/commit/14f904c32a44010d4dfb8a829805648a88c22f53,
since that's the only change that's touched `rapply` lately.

On Wed, Jul 15, 2015 at 12:35 PM, Dayne Filer  wrote:
> In 3.1.2 eval does not store the result of the bquote-generated call in the
> given environment. Interestingly, in 3.2.1 eval does store the result of
> the bquote-generated call in the given environment.
>
> In other words if I run the given example with eval rather than evalq, on
> 3.1.2 "x" is never stored in "fenv," but it is when I run the same code on
> 3.2.1. However, the given example stores "x" in "fenv" on 3.1.2, but throws
> the error I gave when run on 3.2.1.
>
> To give credit, I received the idea for using evalq from SO:
> http://stackoverflow.com/a/22559385
>
> Dayne
>
>
> On Wed, Jul 15, 2015 at 3:29 PM, William Dunlap  wrote:
>
>> I am curious why you used evalq instead of eval in this code.
>>
>> Bill Dunlap
>> TIBCO Software
>> wdunlap tibco.com
>>
>> On Wed, Jul 15, 2015 at 11:49 AM, Dayne Filer 
>> wrote:
>>
>>> Hello,
>>>
>>> I upgraded from 3.1.2 to 3.2.1 and am receiving errors on code that worked
>>> as I intended previously. Briefly, I am using bquote to generate
>>> expressions to modify data.table objects within a  function, so I need the
>>> changes to actually be stored in the given environment. Previously, I used
>>> code like the following:
>>>
>>> test <- list(bquote(x <- 10))
>>> fenv <- environment()
>>> rapply(test, evalq, envir = fenv)
>>>
>>> Although the code in the example above is much simpler, it shows the
>>> problem. On 3.1.2 the expression is evaluated and x is stored as 10 in the
>>> given environment, fenv. In 3.2.1 the code throws an error:
>>>
>>> Error in eval(substitute(expr), envir, enclos) : object 'X' not found
>>>
>>> I could not find anything in the release notes that would explain this
>>> change. Changing evalq to eval works in 3.2.1, but eval does not store x
>>> in
>>> the given environment in 3.1.2.
>>>
>>> Thanks,
>>>
>>> Dayne
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> __
>>> R-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>
>>
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] NEWS.md support on CRAN

[Kevin Ushey]

I like the idea of supporting a small, strict subset of Markdown that
can be used to translate from NEWS.md to NEWS.

Following from Yihui's example, it would be pretty easy to write a
parser in R for such a format (and I'd be willing to try implementing
one, if that would be of interest).

Kevin

On Sun, May 24, 2015 at 10:20 AM, Yihui Xie  wrote:
> That is more or less what I had been doing for a long time (having
> both NEWS.md and NEWS), but decided not to do it any more last year.
> In fact, you can easily convert NEWS.md to a NEWS file that R's news()
> can understand, e.g.
> https://github.com/yihui/knitr/blob/947ad5fc94/Makefile#L8-L10 (if
> your NEWS.md is like this
> https://raw.githubusercontent.com/yihui/knitr/947ad5fc94/NEWS.md)
>
> I stopped doing this because as I said, I found Github release notes
> much more pleasant to read (https://github.com/yihui/knitr/releases),
> and I do not care much about the possibility that some users do not
> have internet connections when reading the NEWS (it is a legitimate
> concern, though). IMHO, it is totally worth it if we are talking about
> official support of Markdown in R's documentation system (.Rd files),
> and it is probably not worth the time and effort if we only want to
> support NEWS.md in particular. That is just a tiny problem compared to
> the effort of porting CommonMark or whatever Markdown rendering
> engines into R.
>
> Regards,
> Yihui
>
>
> On Sun, May 24, 2015 at 7:58 AM, Duncan Murdoch
>  wrote:
>>
>> I imagine GitHub users could have both NEWS.md and NEWS, with one being
>> a symlink to the other, and .Rbuildignore set to ignore NEWS.md.  Why
>> not try it, and post instructions for other Github users?  news() won't
>> be able to understand the headings, but it should display the file as a
>> bunch of text.
>>
>> Duncan Murdoch
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Installation error with R-devel

[Kevin Ushey]

It's hard to diagnose this without your package sources / a
reproducible example.

Shot in the dark: one thing worth checking is that you don't have an
entry in your `.Rbuildignore` that's removing files you don't expect
it to (maybe that's causing R to strip out the 'doc/index.html' file)

Kevin

On Fri, May 15, 2015 at 12:54 PM, Therneau, Terry M., Ph.D.
 wrote:
> I have a local library with functions that interrogates an
> institution-specific web API, so is not of interest to anyone outside of
> Mayo.  For some reason the R CMD INSTALL command fails.  See below:
>
> Build the library, then install it.
>
> tmt-local2127% R CMD build dart
> * checking for file ‘dart/DESCRIPTION’ ... OK
> * preparing ‘dart’:
> * checking DESCRIPTION meta-information ... OK
> * installing the package to build vignettes
> * creating vignettes ... OK
> * checking for LF line-endings in source and make files
> * checking for empty or unneeded directories
> * looking to see if a ‘data/datalist’ file should be added
> * building ‘dart_1.0-6.tar.gz’
>
> tmt-local2128% R CMD INSTALL dart_1.0-6.tar.gz
> * installing to library ‘/home/therneau/Rlib’
> * installing *source* package ‘dart’ ...
> ** R
> ** data
> *** moving datasets to lazyload DB
> ** inst
> ** preparing package for lazy loading
> ** help
> *** installing help indices
> ** building package indices
> ** installing vignettes
> Warning in file(con, "w") :
>   cannot open file '/home/therneau/Rlib/dart/doc/index.html': No such file
> or directory
> Error in file(con, "w") : cannot open the connection
> ERROR: installing vignettes failed
> * removing ‘/home/therneau/Rlib/dart’
>
> -- The installation works ok if I build it with the no-build-vignettes
> option.
> -- If I do that, I can cd to the vignettes directory and Sweave runs fine,
> i.e., library(dart) works
> -- A call to "tar -tf dart_1.0-6.tar.gz" reveals that the inst directory has
> only a NEWS file.
> -- R CMD check fails with the same message
>
> Hints anyone?
> Here is the sessionInfo() data
>
>> sessionInfo()
> R Under development (unstable) (2015-05-14 r68368)
> Platform: i686-pc-linux-gnu (32-bit)
> Running under: Ubuntu 14.04.1 LTS
>
> locale:
>  [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
>  [3] LC_TIME=en_US.UTF-8LC_COLLATE=C
>  [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
>  [7] LC_PAPER=en_US.UTF-8   LC_NAME=C
>  [9] LC_ADDRESS=C   LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods base
>>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Reading exit code of pipe()

[Kevin Ushey]

Hi Jeroen,

I think `pipe` might just be returning the status code of the
underlying command executed; for example, I get a status code of '0'
when I test a pipe on `ls`:

conn <- pipe("ls")
stream <- readLines(conn)
print(close(conn))

Similarly, I get an error code if I try to `ls` a non-existent
directory (512 in my case), e.g.

conn <- pipe("ls /no/path/here/sir")
stream <- readLines(conn)
print(close(conn))

So maybe `cat` just doesn't set a status code, and so there's nothing
for R to forward back (ergo -- NULL)?

Kevin

On Wed, May 13, 2015 at 5:24 PM, Jeroen Ooms  wrote:
> Is there a way to get the status code of a pipe() command? The
> documentation suggests that it might be returned by close, however
> this does not seem to be the case.
>
>   con <- pipe("cat /etc/passwd", "r")
>   stream <- readLines(con, n = 10)
>   err <- close(con)
>   print(err)
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Alternative for wildcard gnu extension in Makevars

[Kevin Ushey]

One other solution that's only a little crazy: you could have a R
function within your package that generates the appropriate (portable)
Makevars, and within the package `configure` script call that
function. For example"

R --vanilla --slave -e "source('R/makevars.R'); makevars()"

And that 'makevars()' function could generate portable
'Makevars(.win)' files for your package.

Kevin

On Wed, May 13, 2015 at 9:08 AM, Gábor Csárdi  wrote:
> On Wed, May 13, 2015 at 12:05 PM, Jan van der Laan 
> wrote:
> [...]
>
>> Too bad. Since it is only a handful of files, I will probably move them
>> directly into the src directory and prefix them. It would have been nice to
>> have been able to keep them separate.
>>
>
> If it is a couple of files, then you can also just list them in SOURCES (or
> even just OBJECTS, with a .o suffix), and leave them where they are.
>
> Gabor
>
> [...]
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Inconsistency when naming a vector

[Kevin Ushey]

In `?names`:

 If ‘value’ is shorter than ‘x’, it is extended by character ‘NA’s
 to the length of ‘x’.

So it is as documented.

That said, it's somewhat surprising that both NA and "" serve as a
placeholder for a 'missing name'; I believe they're treated
identically by R under the hood (e.g. in subsetting operations) but
there may be some subtle cases where they're not.


On Mon, Apr 27, 2015 at 6:08 AM, Suzen, Mehmet  wrote:
>
> There is no inconsistency. Documentation of `names` says "...value
> should be a character vector of up to the same length as x..."
> In the first definition your character vector is not the same length
> as length of x, so you enforce NA by not defining value[2]
>
> x <- 1:2
> value<-c("a")
> value[2]
> [1] NA
>
> where as in the second case, R uses default value "", from `names`
> documentation "..The name "" is special: it is used to indicate that
> there is no name associated with an element.". Since you defined the
> first one, it internally assigns "" to non-defined names to match the
> length of the vector.
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [PATCH] Makefile: add support for git svn clones

[Kevin Ushey]

On Mon, Jan 19, 2015 at 12:34 PM, Felipe Balbi  wrote:
> Hi,
>
> On Mon, Jan 19, 2015 at 03:31:32PM -0500, Duncan Murdoch wrote:
>> >>> git has an interface for cloning SVN repositories into git
>> >>> which some users might decide to use. For those users'
>> >>> surprise, the repository will always fail to build on svnonly
>> >>> target and it will exit early.
>> >>>
>> >>> The problem is simple enough to fix by just checking if a .git
>> >>>  directory exists in top_builddir and, if so, call git svn
>> >>> info insstead of svn info.
>> >>>
>> >>
>> >> I think we are unlikely to accept this change.  Nobody in R Core
>> >> uses git this way, so it would never be tested, and would likely
>> >> soon fail.
>> >
>> > it will be tested by anybody using git svn clone, right ?
>> >
>> >> Indeed, it already fails if someone were to try it on Windows,
>> >> since you didn't patch the makefiles for that platform.
>> >
>> > yeah, sorry about that, I wasn't aware there were windows-specific
>> >  Makefiles with duplicated logic in the repository.
>> >
>> >> The R sources are kept in an SVN repository, and as long as
>> >> that's true, we're only likely to support direct SVN access.
>> >
>> > Fair enough. But don't you think it's a bit odd to couple the
>> > repository compilation with the availability of a specific SCM
>> > tool ?
>> >
>> > I mean, R just won't build unless you have svn info available, I
>> > think that's pretty odd. Printing a warning would be another
>> > possibility, but exitting build is almost an overreaction.
>>
>> That's just false.  Build from a tarball, and you can store it anyway
>> you like.
>
> I'm talking about the SVN repository. Building from a tarball prevents
> me from tracking R's revisions, don't you think ? But as I said, if the
> community doesn't want to support a git svn clone, that's all fine and
> dandy.

The community does; R-Core doesn't. See
https://github.com/wch/r-source/wiki for information on workarounds,
and feel free to add your own workarounds for building R on Windows
from a git checkout (if you have such a thing -- get in touch with
Winston and he'd be happy to modify the wiki)

>
> --
> balbi
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] subscripting a data.frame (without changing row order) changes internal row.names

[Kevin Ushey]

I believe the question here is related to the sign on the compact row
names representation: why is it sometimes `c(NA, )` and
sometimes `c(NA, )` -- why the difference in sign?

To the best of my knowledge, older versions of R used the signed-ness
of compact row.names to differentiate between different 'types' of
data.frames, but that should no longer be necessary. Unless there is
some reason not to, I believe R should standardize on one
representation, and consider it a bug if the other is seen.

Of course, I could be wrong, so I only offer my understanding only as
a way of invoking Cunningham's law...

Cheers,
Kevin

On Mon, Nov 10, 2014 at 12:05 PM, Joshua Ulrich  wrote:
> On Mon, Nov 10, 2014 at 12:35 PM, Dr Gregory Jefferis
>  wrote:
>> Dear R-devel,
>>
>> Can anyone help me to understand this? It seems that subscripting the rows
>> of a data.frame without actually changing their order, somehow changes an
>> internal representation of row.names that is revealed by e.g.
>> dput/dump/serialize
>>
>> I have read the docs and inspected the (R) code for data.frame, rownames,
>> row.names and dput without enlightenment.
>>
> Look at ?.row_names_info (which is mentioned in the See Also section
> of ?row.names) and its type argument.  Also see the discussion here:
> http://stackoverflow.com/q/26468746/271616
>
>> df=data.frame(a=1:10, b=1)
>> dput(df)
>> df2=df[1:nrow(df), ]
>> # R thinks they are equal (so do I!)
>> all.equal(df, df2)
>> dput(df2)
>>
>> Looking at the output of the dputs
>>
>>> dput(df)
>>
>> structure(list(a = 1:10, b = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1)), .Names =
>> c("a",
>> "b"), row.names = c(NA, -10L), class = "data.frame")
>>>
>>> dput(df2)
>>
>> structure(list(a = 1:10, b = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1)), .Names =
>> c("a",
>> "b"), row.names = c(NA, 10L), class = "data.frame")
>>
>> we have row.names = c(NA, -10L) in the first case and row.names = c(NA, 10L)
>> in the second, so somehow these objects have a different representation
>>
>> Can anyone explain why? This has come up because
>>
> The first are "automatic".  The second are a compact form of 1:10, as
> mentioned in ?row.names.  I'm not certain of the root cause/reason,
> but the second object will not have "automatic" rownames because you
> have subset it with a non-missing 'i'.
>
>>> library(digest)
>>> digest(df)==digest(df2)
>>
>> [1] FALSE
>>
>> digest uses serialize under the hood, but serialize, dput and dump all show
>> the same effect (I've pasted an example below using dump, md5sum from base
>> R).
>>
>> Many thanks for any enlightenment! More generally is there any way to
>> calculate a digest of a data.frame that could get round this issue or is
>> that not possible?
>>
>> Best wishes,
>>
>> Greg.
>>
>>
>> A digest using base R:
>>
>> library(tools)
>> td=tempfile()
>> dir.create(td)
>> tempfiles=file.path(td,c("df", "df2"))
>> dump("df",tempfiles[1])
>> dump("df2",tempfiles[2])
>> md5sum(tempfiles)
>>
>> # different md5sum
>>
>>> sessionInfo() # for my laptop but also observed on R 3.1.2
>>
>> R version 3.1.1 (2014-07-10)
>> Platform: x86_64-apple-darwin13.1.0 (64-bit)
>>
>> locale:
>> [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
>>
>> attached base packages:
>> [1] tools stats graphics  grDevices utils datasets  methods
>> base
>>
>> other attached packages:
>> [1] nat_1.5.14  nat.utils_0.4.2 digest_0.6.4Rvcg_0.9
>> devtools_1.6.1  igraph_0.7.1
>> [7] testthat_0.9.1  rgl_0.93.1098
>>
>> loaded via a namespace (and not attached):
>>  [1] codetools_0.2-9   filehash_2.2-2nabor_0.4.3   parallel_3.1.1
>> plyr_1.8.1
>>  [6] Rcpp_0.11.3   rstudio_0.98.1062 rstudioapi_0.1XML_3.98-1.1
>> yaml_2.1.13
>>
>> --
>> Gregory Jefferis, PhD
>> Division of Neurobiology
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue
>> Cambridge Biomedical Campus
>> Cambridge, CB2 OQH, UK
>>
>> http://www2.mrc-lmb.cam.ac.uk/group-leaders/h-to-m/g-jefferis
>> http://jefferislab.org
>> http://flybrain.stanford.edu
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
>
> --
> Joshua Ulrich  |  about.me/joshuaulrich
> FOSS Trading  |  www.fosstrading.com
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] R's internal tar ignores files in sub-directories

[Kevin Ushey]

E.g. I am seeing:

dir <- file.path(tempdir(), "test-tar")
dir.create(dir)
setwd(dir)

dir.create("foo", showWarnings = FALSE)
file.create("foo/bar.R")

tar("test.tar", files = "foo/bar.R")
dir.create("untarred")
untar("test.tar", exdir = "untarred")
list.files("untarred", recursive = TRUE) # character(0)

As far as I can see, the documentation in `?tar` does not reflect that
R's internal `tar` expects paths to directories, not files themselves.
Although I would have preferred the files being included, or at least
a warning / error that they would be excluded.

Thanks,
Kevin

> sessionInfo()
R version 3.1.1 Patched (2014-09-27 r66695)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

loaded via a namespace (and not attached):
[1] tools_3.1.1

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Bug in new behaviour for all.equal and environments?

[Kevin Ushey]

Hi R-devel,

The following code:

all.equal(baseenv(), baseenv())

gives the error when run in a clean R session with latest R-devel (r66650):

kevin:~$ R --vanilla --slave -e "all.equal(baseenv(), baseenv())"
Error in all.equal.envRefClass(target[[i]], current[[i]],
check.attributes = check.attributes,  :
  attempt to apply non-function
Calls: all.equal ... all.equal.list -> all.equal -> all.equal.envRefClass
Execution halted

Although I don't know if it's helpful -- it appears that packages that
include some S4 machinery will effect the outcome of this error, e.g.
if we load 'Biobase' first:

kevin:~$ R --vanilla --slave -e
"suppressPackageStartupMessages(library(Biobase));
all.equal(baseenv(), baseenv())"
Error in target$getClass() : object '.refClassDef' not found
Calls: all.equal ... all.equal.list -> all.equal ->
all.equal.envRefClass -> 
Execution halted

We were bumping into an error with this in Rcpp -- we used all.equal
(through RUnit) to confirm that baseenv(), and a function returning
the base environment, would return TRUE.

For completeness:

kevin:~$ R --vanilla --slave -e "sessionInfo()"
R Under development (unstable) (2014-09-20 r66650)
Platform: x86_64-apple-darwin13.3.0 (64-bit)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] vignette index going AWOL

[Kevin Ushey]

Hi Randall,

I notice that, in your .Rbuildignore, you have the entry:

^build$

and I suspect this is the culprit (having being bitten by a similar
problem before). Some part of the R build / install process creates /
uses that directory, but having it excluded in .Rbuildignore will
cause you grief.

In general, you probably have to consider `build` reserved for use by
R, when developing R packages.

Cheers,
Kevin

On Mon, Aug 25, 2014 at 11:10 AM, Randall Pruim  wrote:
> I’m preparing a package (fastR) for submission to CRAN, but the vignette 
> index keeps going AWOL, or at least R CMD check —as-cran thinks so. I’ve 
> tried several things and gave myself the weekend to think of other things, 
> but I can’t figure it out.  Perhaps someone on the list can lend a hand.
>
> Here’s one example situation, where I build the index.html file myself and 
> put it in inst/doc/index.html
>
> $ cat fastR.Rcheck/00check.log | grep -v OK
> * using log directory ‘/Users/rpruim/projects/github/fastR/fastR.Rcheck’
> * using R Under development (unstable) (2014-08-21 r66456)
> * using platform: x86_64-apple-darwin13.1.0 (64-bit)
> * using session charset: UTF-8
> * checking extension type ... Package
> * this is package ‘fastR’ version ‘0.8-0’
> * checking CRAN incoming feasibility ... NOTE
> Maintainer: ‘Randall Pruim ’
> Package has a VignetteBuilder field but no prebuilt vignette index.
> NOTE: There was 1 note.
>
> $ tar -tf fastR_0.8-0.tar.gz | grep inst/doc
> fastR/inst/doc/
> fastR/inst/doc/Errata.Rnw
> fastR/inst/doc/Errata.pdf
> fastR/inst/doc/Updates.R
> fastR/inst/doc/Updates.Rnw
> fastR/inst/doc/Updates.pdf
> fastR/inst/doc/index.html
>
> There is an index.html in the tar ball.  And If I install the package from 
> the tar ball, the index appears as it should in the package.
>
> I would be happy with the auto-generated index, but that doesn’t seem to be 
> working either.  If I remove index.html I get
>
> $ grep -v OK fastR.Rcheck/00check.log
> * using log directory ‘/Users/rpruim/projects/github/fastR/fastR.Rcheck’
> * using R Under development (unstable) (2014-08-21 r66456)
> * using platform: x86_64-apple-darwin13.1.0 (64-bit)
> * using session charset: UTF-8
> * checking extension type ... Package
> * this is package ‘fastR’ version ‘0.8-1’
> * checking CRAN incoming feasibility ... NOTE
> Maintainer: ‘Randall Pruim ’
> Package has a VignetteBuilder field but no prebuilt vignette index.
> NOTE: There was 1 note.
>
> $ tar -tf fastR_0.8-1.tar.gz | grep inst/doc
> fastR/inst/doc/
> fastR/inst/doc/Errata.Rnw
> fastR/inst/doc/Errata.pdf
> fastR/inst/doc/Updates.R
> fastR/inst/doc/Updates.Rnw
> fastR/inst/doc/Updates.pdf
>
> Comparing with other successful packages created last week, it doesn’t appear 
> that the tar ball contains the vignette index for those either — or perhaps I 
> don’t know what I’m looking for.
>
> $ tar -tf mosaic_0.9.1-3.tar.gz | grep -i index
> mosaic/demo/00Index
>
> (mosaic passes checks fine and are already on CRAN.)
>
> Both packages use knitr to convert Rnw to PDF.  mosaic also uses R.rsp to 
> include static PDFs.
>
> $ grep -i vig  mosaic/DESCRIPTION
> VignetteBuilder: knitr, R.rsp
>
> [09:22][rpruim@Mac21521:~/projects/github]
> $ grep -i vig  fastR/DESCRIPTION
> VignetteBuilder: knitr
>
> $ grep  Vig mosaic/vignettes/*Rnw
> %\VignetteIndexEntry{mosaic resources}
> %\VignettePackage{mosaic}
> %\VignetteKeywords{mosaic, vignettes, resources, references}
> %\VignetteEngine{knitr::knitr}
>
> $ grep  Vig fastR/vignettes/*Rnw
> fastR/vignettes/Errata.Rnw:%\VignetteEngine{knitr::knitr}
> fastR/vignettes/Errata.Rnw:%\VignetteIndexEntry{Errata}
> fastR/vignettes/Errata.Rnw:%\VignettePackage{fastR}
> fastR/vignettes/Errata.Rnw:%\VignetteKeywords{fastR, vignettes}
> fastR/vignettes/Updates.Rnw:%\VignetteEngine{knitr::knitr}
> fastR/vignettes/Updates.Rnw:%\VignetteIndexEntry{Updates to R Programming in 
> FASt}
> fastR/vignettes/Updates.Rnw:%\VignettePackage{fastR}
> fastR/vignettes/Updates.Rnw:%\VignetteKeywords{mosaic, fastR, vignettes, 
> resources}
>
> There must be some subtle difference between my two packages that I just 
> can’t spot.
>
> Thanks in advance for any pointers about where to look.
>
> —rjp
>
> PS.  The package is on github and installs fine (vignette index included, 
> even though there is no index.html in the repo) using install_github("fastR", 
> "rpruim")
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Removed 'extern' on R_running_as_main_program in Rinterface.h -- intentional?

[Kevin Ushey]

Hi R-devel,

In this commit:
https://github.com/wch/r-source/commit/c67a107dd7b0f074cec9359b8e2ca07c6243283c

R_running_as_main_program was moved from Rmain.c to Rinterface.h, and lost
its 'extern' declaration. This change is causing us linker problems (since
we now have duplicate symbols for R_running_as_main_program). Was this
intentional? If so, what else should applications embedding R be doing to
indicate they are running as the main program?

In addition, I guess the documentation at
http://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Embedding-R-under-Unix_002dalikes
should be updated to reflect the change.

Thanks,
Kevin

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] USE_CXX1X, Snow Leopard R binaries + Mavericks

[Kevin Ushey]

Hi R-devel,

I'm noticing the following behaviour:

writeLines("#include ", file = "test.cpp")
Rcpp::sourceCpp("~/test.cpp") ## succeeds at trivial compile
Sys.setenv("USE_CXX1X" = "yes")
Rcpp::sourceCpp("~/test.cpp") ## fails; CXX nor CXX1X properly set (?)

IIUC, R is not propagating CXX nor CXX1X when USE_CXX1X is set on a
Snow Leopard CRAN build of R. Is this the expected behaviour? IIUC,
Snow Leopard binaries are built with older compilers and hence we
might not expect this to even work were CXX or CXX1X propagated (ie,
grabbing the Mavericks clang / clang++).

Comments on this in R-exts would be greatly appreciated as well.

kevinushey@s-169-232-64-218:~$ R --vanilla --slave -e "sessionInfo()"
R version 3.1.0 (2014-04-10)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

kevinushey@s-169-232-64-218:~$ uname -a

Darwin s-169-232-64-218.resnet.ucla.edu 13.2.0 Darwin Kernel Version
13.2.0: Thu Apr 17 23:03:13 PDT 2014;
root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64

Cheers from useR2014,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] operation on ‘numsels’ may be undefined

[Kevin Ushey]

Hi Hervé,

Glad you enjoyed the slides :)

Here's a bit more material discussing why 'i = ++i' and friends are
actually defined under the C++11 standard, but undefined before:

Why it might be well-defined in C++0x and above:

http://stackoverflow.com/questions/3852768/unsequenced-value-computations-a-k-a-sequence-points

Why i = i++ is undefined, and i = ++i is defined, for C++0x and above:

http://stackoverflow.com/questions/17400137/order-of-evaluation-and-undefined-behaviour

http://stackoverflow.com/questions/3690141/multiple-preincrement-operations-on-a-variable-in-cc/3691469#3691469

I would imagine this would translate into a similar undefined-ness for the
C standards, but would not be surprised if I were wrong... IIUC, though,
using the rules that

1. A variable may only be modified once between two sequence points, and
2. 'i = ++i' modifies 'i' twice without a sequence point to separate
them

means it's undefined there. IIUC (and I may not), precedence doesn't even
play in here -- if we modify a variable twice between two sequence points,
we have UB.

Anyway, I think I've gone too far down the rabbit hole into the language
lawyer land of C/C++ standards, so let's all just agree that, regardless of
what the correct interpretation is, 'i = i++' and 'i = ++i' are just things
you shouldn't write :)

Cheers,
Kevin


On Tue, Jun 24, 2014 at 3:38 PM, Hervé Pagès  wrote:

> Hi Kevin,
>
>
> On 06/23/2014 10:28 PM, Kevin Ushey wrote:
>
>> I don't see what's so surprising here.
>>
>> That statement is identical to writing:
>>
>>  if (arrMask[i] == 1) {
>>  numsels = ++numsels;
>>  } else {
>>  numsels = numsels;
>>  }
>>
>> and
>>
>>  numsels = ++numsels;
>>
>> has two statements modifying the value of numsels (= and prefix-++) in
>> a single sequence point. (Do we increment then assign, or assign then
>> increment? The C / C++ standards leave this undefined.)
>>
>> Imagine writing the operations out as functions: we have the `=`
>> function, and the `prefix-++` function -- both of these 'mo dify' (one
>>
>> of) their arguments. Do we evaluate it as `=`(a, `prefix-++`(a)) or
>> `prefix-++`(`=`(a, a))?
>>
>
> As the former? Isn't it that `prefix-++` has precedence over `=`? Also
> according to the standard, `prefix-++` must perform the increment
> *before* returning the incremented value. So I'm still a little bit
> confused about this warning. Although I would understand it if
> `suffix-++` was used instead of `prefix-++`:
>
> numsels = numsels++;
>
> In that case the incrementation happens *after* the right value
> of the assignment is evaluated but it's undefined whether it will
> happen before or after the assignment itself (on my machine it
> happens after).
>
>
>  The C standard leaves this undefined, so
>> compilers are free to do what they wish (and the nice ones warn you
>> when there is such an ambiguity). I guess the net result of the
>> operation is the same in each case _here_, but this is of course not
>> the case for other functions that modify the value of their
>> operand(s). And, in truth, this is _undefined behaviour_ and so the
>> compiler could still rightly make demons fly out of your nose if it
>> wanted to upon program execution.
>>
>> I highly recommend reading the slides at
>> http://www.slideshare.net/olvemaudal/deep-c, especially the bit on
>> sequence points.
>>
>
> Nice slides. Pretty entertaining! The numsels = numsels++ case is
> actually discussed at slide 142. Thanks for the link!
>
>
> H.
>
>
>> Cheers,
>> Kevin
>>
>> On Mon, Jun 23, 2014 at 9:22 PM, Kasper Daniel Hansen
>>  wrote:
>>
>>> I am not an expert on this, but I note that the section on
>>> -Wsequence-point
>>> at
>>>http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
>>> specifically mentions ? and :.  Perhaps some more work on tracking down
>>> their definitions and precedence might lead to insights.
>>>
>>> Best,
>>> Kasper
>>>
>>>
>>> On Mon, Jun 23, 2014 at 6:42 PM, Hervé Pagès  wrote:
>>>
>>>  On 06/23/2014 03:18 PM, Hervé Pagès wrote:
>>>>
>>>>  Hi Christian,
>>>>>
>>>>> On 06/23/2014 11:54 AM, cstrato wrote:
>>>>>
>>>>>  Dear Romain,
>>>>>>
>>>>>> I do not know enough about compilers, but as far as I remember, they
>>>>>> 'wor

Re: [Rd] Curious behavior of $ and lapply

[Kevin Ushey]

`lapply` basically takes its call and massages into calls of the following form:

FUN(X[[1L]], ...)
FUN(X[[2L]], ...)
... 

that get evaluated in the appropriate environment.

For `lapply(list(list(a=3,b=4)),"$","b")`, you can imagine that a
function `FUN` of the form:

FUN <- function(x, ...) "$"(x, ...)

is being generated, and then evaluated as

FUN(list(list(a=3, b=4))[[1L]], "b") ## returns NULL

and I can only guess that the non-standard evaluation of `$` is not
interpreting `...` as you expect.

Moral of the story -- lapply does non-standard evaluation, and does
not compose nicely with other functions performing non-standard
evaluation. This is discussed a bit in the Note: section of `?lapply`.
Copied from there (emphasis mine):

For historical reasons, the calls created by lapply are unevaluated,
and code has been written (e.g. bquote) that relies on this. This
means that the recorded call is always of the form FUN(X[[i]], ...),
with i replaced by the current (integer or double) index. This is not
normally a problem, but it can be if FUN uses sys.call or match.call
or if it __is a primitive function that makes use of the call__. This
means that it is often safer to call primitive functions with a
wrapper, so that e.g. lapply(ll, function(x) is.numeric(x)) is
required to ensure that method dispatch for is.numeric occurs
correctly.

Cheers,
Kevin


On Mon, Jun 23, 2014 at 3:04 PM, Stavros Macrakis (Σταῦρος Μακράκης)
 wrote:
> There seems to be a funny interaction between lapply and "$" -- also, "$"
> doesn't signal an error in some cases where "[[" does.
>
> The $ operator accepts a string second argument in functional form:
>
>> `$`(list(a=3,b=4),"b")
> [1] 4
>
> lapply(list(list(a=3,b=4)),function(x) `$`(x,"b"))
> [[1]]
> [1] 4
>
> ... but using an lapply "..." argument, this returns NULL:
>
>> lapply(list(list(a=3,b=4)),"$","b")
> [[1]]
> NULL
>
> Equally strangely, a symbol is an error in functional form:
>
>>  `$`(list(a=3,b=4),quote(`b`))
> Error in list(a = 3, b = 4)$quote(b) : invalid subscript type 'language'
>
> ... but not an error in lapply, where it returns NULL:
>
>> lapply(list(list(a=3,b=4)),`$`,quote(`b`))
> [[1]]
> NULL
>
> I wonder if lapply is failing to pass the second argument to $ correctly,
> since $ with various unreasonable second arguments returns NULL (shouldn't
> it given an error?):
>
> "$"(list(list(a=3,b=4)),) => NULL
> lapply(list(list(a=3,b=4)),`$`,function(i)i) => NULL
>
> Compare
>
> "[["(list(list(a=3,b=4)),) => <>
>
> But with [[, lapply correctly gives an error:
>
> lapply(list(list(a=3,b=4)),`[[`,function(i)i) => <>
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] operation on ‘numsels’ may be undefined

[Kevin Ushey]

I don't see what's so surprising here.

That statement is identical to writing:

if (arrMask[i] == 1) {
numsels = ++numsels;
} else {
numsels = numsels;
}

and

numsels = ++numsels;

has two statements modifying the value of numsels (= and prefix-++) in
a single sequence point. (Do we increment then assign, or assign then
increment? The C / C++ standards leave this undefined.)

Imagine writing the operations out as functions: we have the `=`
function, and the `prefix-++` function -- both of these 'modify' (one
of) their arguments. Do we evaluate it as `=`(a, `prefix-++`(a)) or
`prefix-++`(`=`(a, a))? The C standard leaves this undefined, so
compilers are free to do what they wish (and the nice ones warn you
when there is such an ambiguity). I guess the net result of the
operation is the same in each case _here_, but this is of course not
the case for other functions that modify the value of their
operand(s). And, in truth, this is _undefined behaviour_ and so the
compiler could still rightly make demons fly out of your nose if it
wanted to upon program execution.

I highly recommend reading the slides at
http://www.slideshare.net/olvemaudal/deep-c, especially the bit on
sequence points.

Cheers,
Kevin

On Mon, Jun 23, 2014 at 9:22 PM, Kasper Daniel Hansen
 wrote:
> I am not an expert on this, but I note that the section on -Wsequence-point
> at
>   http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
> specifically mentions ? and :.  Perhaps some more work on tracking down
> their definitions and precedence might lead to insights.
>
> Best,
> Kasper
>
>
> On Mon, Jun 23, 2014 at 6:42 PM, Hervé Pagès  wrote:
>
>> On 06/23/2014 03:18 PM, Hervé Pagès wrote:
>>
>>> Hi Christian,
>>>
>>> On 06/23/2014 11:54 AM, cstrato wrote:
>>>
 Dear Romain,

 I do not know enough about compilers, but as far as I remember, they
 'work' from right to left,

>>>
>>> Not necessarily. So you should not rely on that. This is what the
>>> (somewhat obscure) warning you see on zin1 is trying to tell you.
>>>
>>
>> Actually, I don't see an ambiguity in your code:
>>
>>
>>   numsels = (arrMask[i] == 1) ? ++numsels : numsels;
>>
>> Yes it's confusing and unnecessarily complicated but I don't see that
>> it relies on some undefined behavior. It's not like in the thread on
>> Bioc-devel where the expression was:
>>
>>   *p++ = tolower(*p);
>>
>> In that case the left-value of the assignment is itself an expression
>> that needs to be evaluated and the outcome of the assignment depends
>> on whether the left-value is evaluated before the right expression or
>> not. But in your case the left-value is a variable name so there is
>> nothing to evaluate.
>>
>> So I don't know. Looks like a false positive from the gcc compiler to
>> me. Someone on the list might have a better insight.
>>
>> Cheers,
>>
>> H.
>>
>>
>>  Personally I would just do:
>>>
>>>  if (arrMask[i] == 1) numsels++;
>>>
>>> which is the standard way to implement the "if (some condition)
>>> then increment counter" idiom.
>>>
>>> As Kasper mentioned, a similar issue was recently discussed here:
>>>
>>>https://stat.ethz.ch/pipermail/bioc-devel/2014-June/005858.html
>>>
>>> Cheers,
>>> H.
>>>
>>>
>>>  so numsels = ++numsels should not confuse the
 compiler. Anyhow I will change my code to your first suggestion since it
 is more elegant.

 Best regards,
 Christian


 On 6/23/14 7:13 PM, Romain François wrote:

>
> Le 23 juin 2014 à 18:28, cstrato  a écrit :
>
>  Dear Romain,
>>
>> Thank you for your suggestions, I like especially the first one.
>>
>> However, you did not explain why I have never got this warning
>> message on any compiler, and why only one of the two identical Ubuntu
>> compilers did give this warning message?
>>
>> Best regards,
>> Christian
>>
>
> I don’t know, but this:
>
> numsels = ++numsels ;
>
> seems fishy to me, and so it keeps feeling weird with the addition of
> the ternary operator.
>
> There is obviously a difference of setup between these two machines,
> but I don’t have time to sherlock that for you. One of the compilers
> is getting more careful than the other. Getting warnings you did not
> get before is a good thing, as it helps you update the code with that
> new insight.
>
> Welcome to my world, I’m sometimes thrown all kinds of new warnings
> from esoteric compilers, all of them have value .
>
> Romain
>
>  On 6/23/14 3:45 PM, Romain François wrote:
>>
>>>
>>> Le 23 juin 2014 à 15:20, cstrato  a écrit :
>>>
>>>  Dear all,

 Since many years the following C++ code does compile on ALL
 Bioconductor servers (Linux, Windows, Mac) without any warnings:

Int_t numsels = 0;  //number of selected entries
...
for (Int_t i=0; i>>> 

Re: [Rd] How to test if an object/argument is "parse tree" - without evaluating it?

[Kevin Ushey]

Henrik,

If I understand correctly, you want something along the lines of
(following your example):

foo <- function(expr) {
  if (!is.language(expr)) substitute(expr)
  else expr
}

## first example
expr0 <- foo({ x <- 1 })
expr1 <- foo(expr0)
stopifnot(identical(expr1, expr0))

## second
expr2 <- foo(foo(foo(foo(expr0
stopifnot(identical(expr2, expr0))

Hadley's guide on NSE + language elements in R
(http://adv-r.had.co.nz/Computing-on-the-language.html,
http://adv-r.had.co.nz/Expressions.html) may be helpful here.

Cheers,
Kevin

On Thu, May 1, 2014 at 1:54 PM, Duncan Murdoch  wrote:
> On 01/05/2014, 4:39 PM, Henrik Bengtsson wrote:
>>
>> This may have been asked before, but is there an elegant way to check
>> whether an variable/argument passed to a function is a "parse tree"
>> for an (unevaluated) expression or not, *without* evaluating it if
>> not?
>
>
> "Parse tree" isn't R terminology.  Could you give an example of one call
> that passes a parse tree, and one that doesn't?
>
> Duncan Murdoch
>
>
>>
>> Currently, I do various rather ad hoc eval()+substitute() tricks for
>> this that most likely only work under certain circumstances. Ideally,
>> I'm looking for a isParseTree() function such that I can call:
>>
>> expr0 <- foo({ x <- 1 })
>> expr1 <- foo(expr0)
>> stopifnot(identical(expr1, expr0))
>>
>> where foo() is:
>>
>> foo <- function(expr) {
>>if (!isParseTree(expr))
>>  expr <- substitute(expr)
>>expr
>> }
>>
>> I also want to be able to do:
>>
>> expr2 <- foo(foo(foo(foo(expr0
>> stopifnot(identical(expr2, expr0))
>>
>> and calling foo() from within other functions that may use the same
>> "tricks".  The alternative is of course to do:
>>
>> foo <- function(expr, substitute=TRUE) {
>>if (substitute) expr <- substitute(expr)
>>expr
>> }
>>
>> but it would be neat to do this without passing an extra argument.  If
>> this is not possible to implement in plain R, can this be done
>> internally inspecting SEXP:s and so on?  Even better would be if
>> substitute() could do this for me, e.g.
>>
>> expr <- substitute(expr, unlessAlreadyDone=TRUE)
>>
>> Any suggestions?
>>
>> Thanks,
>>
>> Henrik
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Is it possible to shrink an R object in place?

[Kevin Ushey]

Suppose I generate an integer vector with e.g.

SEXP iv = PROTECT(allocVector(INTSXP, 100));

and later want to shrink the object, e.g.

shrink(iv, 50);

would simply re-set the length to 50, and allow R to reclaim the
memory that was previously used.

Is it possible to do this while respecting how R manages memory?

The motivation: there are many operations where the length of the
output is not known ahead of time, and in such cases one typically
uses a data structure that can grow efficiently. Unfortunately, IIUC
SEXPRECs cannot do this; however, an alternative possibility would
involve reserving extra memory, and then shrinking to fit after the
operation is complete.

There have been some discussions previously that defaulted to answers
of the form "you should probably just copy", e.g.
https://stat.ethz.ch/pipermail/r-devel/2008-March/048593.html, but I
wanted to ping and see if others had ideas, or if perhaps there was
code in the R sources that might be relevant.

Another reason why this is interesting is due to C++11 and
multi-threading: if I can pre-allocate SEXPs that will contain results
in the main thread, and then fill these SEXPs asynchronously (without
touching R, and hence not getting in the way of the GC or otherwise),
I can then fill these SEXPs in place and shrink-to-fit after the
computations have been completed. With C++11 support coming with R
3.1.0, functionality like this is very attractive.

The obvious alternatives are to 1) determine the length of the output
first and hence generate SEXPs of appropriate size right off the bat
(potentially expensive), and 2) fill thread-safe containers and copy
to an R object (definitely expensive).

I am probably missing something subtle (or obvious) as to why this may
not work, or be recommended, so I appreciate any comments.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] C API to get numrow of data frame

[Kevin Ushey]

The safest way is to check the length of the row.names attribute, e.g.

length(getAttrib(df, R_RowNamesSymbol)).

This protects you from both data.frames with zero columns, as well as
corrupted data.frames containing columns with different lengths, since
by definition the number of rows in a data.frame is defined by its
row.names attribute. However, R will internally un-collapse a
collapsed row.names on this getAttrib call, which is probably
undesired for very large data.frames.

One way of getting around this is calling .row_names_info from R, e.g.
(modulo my errors):

int df_nrows(SEXP s) {
if (!Rf_inherits(s, "data.frame")) Rf_error("expecting a data.frame");
SEXP two = PROTECT(Rf_ScalarInteger(2));
SEXP call = PROTECT( Rf_lang3(
  Rf_install(".row_names_info"),
  s,
  two
) );
SEXP result = PROTECT(Rf_eval(call, R_BaseEnv));
int output = INTEGER(result)[0];
UNPROTECT(3);
return output;
}

More ideally (?), such a function could be added to util.c and
exported by R, e.g. (again, modulo my errors):

int df_nrows(SEXP s) {
if (!inherits(s, "data.frame")) error("expecting a data.frame");
SEXP t = getAttrib0(s, R_RowNamesSymbol);
if (isInteger(t) && INTEGER(t)[0] == NA_INTEGER && LENGTH(t) == 2)
  return abs(INTEGER(t)[1]);
else
  return LENGTH(t);
}

or even incorporated into the already available 'nrows' function.
Although there is probably someone out there depending on 'nrows'
returning the number of columns for their data.frame...

Cheers,
Kevin

On Mon, Mar 31, 2014 at 6:27 PM, Murray Stokely  wrote:
> I didn't look at the names because I believe that would be incorrect
> if the row names were stored internally in the compact form.
>
> See ?.set_row_names (hat tip, Tim Hesterberg who showed me this years ago) :
>
>  'row.names' can be stored internally in compact form.
>  '.set_row_names(n)' generates that form for automatic row names of
>  length 'n', to be assigned to 'attr(, "row.names")'.
>  '.row_names_info' gives information on the internal form of the
>  row names for a data frame: for details of what information see
>  the argument 'type'.
>
> The function I wrote obviously doesn't work for 0 row or 0 column
> data.frames, you need to check for that.
>
> On Mon, Mar 31, 2014 at 6:12 PM, Gábor Csárdi  wrote:
>> I think it is actually better to check the length of the row names. In case
>> the data frame has zero columns. (FIXME, of course.)
>>
>> Gabor
>>
>>
>> On Mon, Mar 31, 2014 at 8:04 PM, Murray Stokely  wrote:
>>>
>>> The simplest case would be:
>>>
>>>int num_rows = Rf_length(VECTOR_ELT(dataframe, 0));
>>>int num_columns = Rf_length(dataframe);
>>>
>>> There may be edge cases for which this doesn't work; would need to
>>> look into how the dim primitive is implemented to be sure.
>>>
>>>- Murray
>>>
>>>
>>> On Mon, Mar 31, 2014 at 4:40 PM, Sandip Nandi 
>>> wrote:
>>> > Hi ,
>>> >
>>> > Is there any C API to the R API  nrow of dataframe ?
>>> >
>>> > x<- data.frame()
>>> > n<- nrow(x)
>>> > print(n)
>>> > 0
>>> >
>>> >
>>> > Example :
>>> > My C function which deals with data frame looks like and I don't to send
>>> > the  number of rows of data frame .I want to detect it from the function
>>> > itself, my function take data frame as argument and do some on it. I
>>> > want
>>> > API equivalent to nrow. I tried Rf_nrows,Rf_ncols . No much help.
>>> >
>>> > SEXP  writeRR(SEXP dataframe) {
>>> >
>>> > }
>>> >
>>> >
>>> > Any help is very appreciated.
>>> >
>>> > Thanks,
>>> > Sandip
>>> >
>>> > [[alternative HTML version deleted]]
>>> >
>>> > __
>>> > R-devel@r-project.org mailing list
>>> > https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>> __
>>> R-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] internal string comparison (Scollate)

[Kevin Ushey]

I too think it would be useful if R exported some version of its
string sorting routines, since sorting strings while respecting
locale, and doing so in a portable fashion while respecting the user's
environment, is not trivial. R holds a fast, portable, well-tested
solution, and I think package developers would be very appreciative if
some portion of this was exposed at the C level.

If not `Scollate`, then perhaps other candidates could be the more
generic `sortVector`, or the more string-specific (and NA-respecting)
`scmp`.

I understand that the volunteers at R Core have limited time and
resources, and exposing an API imposes additional maintenance burdens
on an already thinly stretched team, but this is a situation where the
R users and package authors alike could benefit. Or, if there are
other reasons why exporting such routines is not possible nor
recommended, it would be very informative to know why.

Thanks,
Kevin

On Thu, Mar 27, 2014 at 11:08 AM, Dirk Eddelbuettel  wrote:
>
> On 26 March 2014 at 19:09, Romain François wrote:
> | That's one part of the problem. Indeed I'd rather use something rather than
> | copy and paste it and run the risk of being outdated. The answer to that is
>
> We all would. But "they" won't let us by refusing to create more API access 
> points.
>
> | testing though. I can develop a test suite that can let me know I'm out of
>
> Correct.
>
> | date and I need to copy and paste some new code, etc ... Done that before, 
> this
> | is tedious, but so what.
> |
> | The other part of the problem (the real part of the problem actually) is 
> that,
> | at least when R is built with ICU support, Scollate will depend on a the
> | collator pointer in util.c
> | https://github.com/wch/r-source/blob/trunk/src/main/util.c#L1777
> |
> | And this can be controlled by the base::icuSetCollate function. Of course 
> the
> | collator pointer is not public.
>
> So the next (and even less pleasant) answer is to build a new package which
> links to, (or worse yet, embeds) libicu.
>
> As you want ICU behaviour, you will need ICU code.
>
> Dirk
>
> --
> Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Question re: NA, NaNs in R

[Kevin Ushey]

Hi Duncan,

Thanks a ton -- I appreciate your taking the time to investigate this,
and especially even checking into the IEEE standard to clarify.

Cheers,
Kevin

On Mon, Feb 10, 2014 at 11:54 AM, Rainer M Krug  wrote:
>
>
> On 02/10/14, 19:07 , Duncan Murdoch wrote:
>> On 10/02/2014 10:21 AM, Tim Hesterberg wrote:
>>> This isn't quite what you were asking, but might inform your choice.
>>>
>>> R doesn't try to maintain the distinction between NA and NaN when
>>> doing calculations, e.g.:
>>> > NA + NaN
>>> [1] NA
>>> > NaN + NA
>>> [1] NaN
>>> So for the aggregate package, I didn't attempt to treat them differently.
>>
>> This looks like a bug to me.  In 32 bit 3.0.2 and R-patched I see
>>
>>> NA + NaN
>> [1] NA
>>> NaN + NA
>> [1] NA
>
> But under 3.0.2 patched 64 bit on Maverick:
>
>> version
>_
> platform   x86_64-apple-darwin10.8.0
> arch   x86_64
> os darwin10.8.0
> system x86_64, darwin10.8.0
> status Patched
> major  3
> minor  0.2
> year   2014
> month  01
> day07
> svn rev64692
> language   R
> version.string R version 3.0.2 Patched (2014-01-07 r64692)
> nickname   Frisbee Sailing
>> NA+NaN
> [1] NA
>> NaN+NA
> [1] NaN
>
>>
>> This seems more reasonable to me.  NA should propagate.  (I can see an
>> argument for NaN for the answer here, as I can't think of any possible
>> non-missing value that would give anything else when added to NaN, but
>> the answer should not depend on the order of operands.)
>>
>> However, I get the same as you in 64 bit 3.0.2.  All calculations I've
>> shown are on 64 bit Windows 7.
>>
>> Duncan Murdoch
>>
>>
>>>
>>> The aggregate package is available at
>>> http://www.timhesterberg.net/r-packages
>>>
>>> Here is the inst/doc/missingValues.txt file from that package:
>>>
>>> --
>>> Copyright 2012 Google Inc. All Rights Reserved.
>>> Author: Tim Hesterberg 
>>> Distributed under GPL 2 or later.
>>>
>>>
>>> Handling of missing values and not-a-numbers.
>>>
>>>
>>> Here I'll note how this package handles missing values.
>>> I do it the way R handles them, rather than the more strict way that
>>> S+ does.
>>>
>>> First, for terminology,
>>>NaN = "not-a-number", e.g. the result of 0/0
>>>NA  = "missing value" or "true missing value", e.g. survey
>>> non-response
>>>xx  = I'll uses this for the union of those, or "missing value of
>>> any kind".
>>>
>>> For background, at the hardware level there is an IEEE standard that
>>> specifies that certain bit patterns are NaN, and specifies that
>>> operations involving an NaN result in another NaN.
>>>
>>> That standard doesn't say anything about missing values, which are
>>> important in statistics.
>>>
>>> So what R and S+ do is to pick one of the bit patterns and declare
>>> that to be a NA.  In other words, the NA bit pattern is a subset of
>>> the NaN bit patterns.
>>>
>>> At the user level, the reverse seems to hold.
>>> You can assign either NA or NaN to an object.
>>> But:
>>> is.na(x) returns TRUE for both
>>> is.nan(x) returns TRUE for NaN and FALSE for NA
>>> Based on that, you'd think that NaN is a subset of NA.
>>> To tell whether something is a true missing value do:
>>> (is.na(x) & !is.nan(x))
>>>
>>> The S+ convention is that any operation involving NA results in an NA;
>>> otherwise any operation involving NaN results in NaN.
>>>
>>> The R convention is that any operation involving xx results in an xx;
>>> a missing value of any kind results in another missing value of any
>>> kind.  R considers NA and NaN equivalent for testing purposes:
>>> all.equal(NA_real_, NaN)
>>> gives TRUE.
>>>
>>> Some R functions follow the S+ convention, e.g. the Math2 functions
>>> in src/main/arithmetic.c use this macro:
>>> #define if_NA_Math2_set(y,a,b)\
>>>     if  (ISNA (a) || ISNA (b)) y = NA_REAL;\
>>> else if (ISNAN(a) || ISNAN(b)) y = R_NaN;
>>>
>>> Other R functions, like the

Re: [Rd] Question re: NA, NaNs in R

[Kevin Ushey]

> else if (ISNAN(a) || ISNAN(b)) y = R_NaN;
>>
>> Other R functions, like the basic arithmetic operations +-/*^,
>> do not (search for PLUSOP in src/main/arithmetic.c).
>> They just let the hardware do the calculations.
>> As a result, you can get odd results like
>> > is.nan(NA_real_ + NaN)
>> [1] FALSE
>> > is.nan(NaN + NA_real_)
>> [1] TRUE
>>
>> The R help files help(is.na) and help(is.nan) suggest that
>> computations involving NA and NaN are indeterminate.
>>
>> It is faster to use the R convention; most operations are just
>> handled by the hardware, without extra work.
>>
>> In cases like sum(x, na.rm=TRUE), the help file specifies that both NA
>> and NaN are removed.
>>
>>
>>
>>
>> >There is one NA but mulitple NaNs.
>> >
>> >And please re-read 'man memcmp': your cast is wrong.
>> >
>> >On 10/02/2014 06:52, Kevin Ushey wrote:
>> >> Hi R-devel,
>> >>
>> >> I have a question about the differentiation between NA and NaN values
>> >> as implemented in R. In arithmetic.c, we have
>> >>
>> >> int R_IsNA(double x)
>> >> {
>> >>  if (isnan(x)) {
>> >> ieee_double y;
>> >> y.value = x;
>> >> return (y.word[lw] == 1954);
>> >>  }
>> >>  return 0;
>> >> }
>> >>
>> >> ieee_double is just used for type punning so we can check the final
>> >> bits and see if they're equal to 1954; if they are, x is NA, if
>> >> they're not, x is NaN (as defined for R_IsNaN).
>> >>
>> >> My question is -- I can see a substantial increase in speed (on my
>> >> computer, in certain cases) if I replace this check with
>> >>
>> >> int R_IsNA(double x)
>> >> {
>> >>  return memcmp(
>> >>  (char*)(&x),
>> >>  (char*)(&NA_REAL),
>> >>  sizeof(double)
>> >>  ) == 0;
>> >> }
>> >>
>> >> IIUC, there is only one bit pattern used to encode R NA values, so
>> >> this should be safe. But I would like to be sure:
>> >>
>> >> Is there any guarantee that the different functions in R would return
>> >> NA as identical to the bit pattern defined for NA_REAL, for a given
>> >> architecture? Similarly for NaN value(s) and R_NaN?
>> >>
>> >> My guess is that it is possible some functions used internally by R
>> >> might encode NaN values differently; ie, setting the lower word to a
>> >> value different than 1954 (hence being NaN, but potentially not
>> >> identical to R_NaN), or perhaps this is architecture-dependent.
>> >> However, NA should be one specific bit pattern (?). And, I wonder if
>> >> there is any guarantee that the different functions used in R would
>> >> return an NaN value as identical to R_NaN (which appears to be the
>> >> 'IEEE NaN')?
>> >>
>> >> (interested parties can see + run a simple benchmark from the gist at
>> >> https://gist.github.com/kevinushey/8911432)
>> >>
>> >> Thanks,
>> >> Kevin
>> >>
>> >> __
>> >> R-devel@r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-devel
>> >>
>> >
>> >
>> >--
>> >Brian D. Ripley,  rip...@stats.ox.ac.uk
>> >Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>> >University of Oxford, Tel:  +44 1865 272861 (self)
>> >1 South Parks Road, +44 1865 272866 (PA)
>> >Oxford OX1 3TG, UKFax:  +44 1865 272595
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Question re: NA, NaNs in R

[Kevin Ushey]

Thanks Tim, this is exactly the explanation I was hoping to see. Much
appreciated!

On Mon, Feb 10, 2014 at 7:21 AM, Tim Hesterberg  wrote:
> This isn't quite what you were asking, but might inform your choice.
>
> R doesn't try to maintain the distinction between NA and NaN when
> doing calculations, e.g.:
>> NA + NaN
> [1] NA
>> NaN + NA
> [1] NaN
> So for the aggregate package, I didn't attempt to treat them differently.
>
> The aggregate package is available at
> http://www.timhesterberg.net/r-packages
>
> Here is the inst/doc/missingValues.txt file from that package:
>
> --
> Copyright 2012 Google Inc. All Rights Reserved.
> Author: Tim Hesterberg 
> Distributed under GPL 2 or later.
>
>
> Handling of missing values and not-a-numbers.
>
>
> Here I'll note how this package handles missing values.
> I do it the way R handles them, rather than the more strict way that S+ does.
>
> First, for terminology,
>   NaN = "not-a-number", e.g. the result of 0/0
>   NA  = "missing value" or "true missing value", e.g. survey non-response
>   xx  = I'll uses this for the union of those, or "missing value of any kind".
>
> For background, at the hardware level there is an IEEE standard that
> specifies that certain bit patterns are NaN, and specifies that
> operations involving an NaN result in another NaN.
>
> That standard doesn't say anything about missing values, which are
> important in statistics.
>
> So what R and S+ do is to pick one of the bit patterns and declare
> that to be a NA.  In other words, the NA bit pattern is a subset of
> the NaN bit patterns.
>
> At the user level, the reverse seems to hold.
> You can assign either NA or NaN to an object.
> But:
> is.na(x) returns TRUE for both
> is.nan(x) returns TRUE for NaN and FALSE for NA
> Based on that, you'd think that NaN is a subset of NA.
> To tell whether something is a true missing value do:
> (is.na(x) & !is.nan(x))
>
> The S+ convention is that any operation involving NA results in an NA;
> otherwise any operation involving NaN results in NaN.
>
> The R convention is that any operation involving xx results in an xx;
> a missing value of any kind results in another missing value of any
> kind.  R considers NA and NaN equivalent for testing purposes:
> all.equal(NA_real_, NaN)
> gives TRUE.
>
> Some R functions follow the S+ convention, e.g. the Math2 functions
> in src/main/arithmetic.c use this macro:
> #define if_NA_Math2_set(y,a,b)  \
> if  (ISNA (a) || ISNA (b)) y = NA_REAL; \
> else if (ISNAN(a) || ISNAN(b)) y = R_NaN;
>
> Other R functions, like the basic arithmetic operations +-/*^,
> do not (search for PLUSOP in src/main/arithmetic.c).
> They just let the hardware do the calculations.
> As a result, you can get odd results like
>> is.nan(NA_real_ + NaN)
> [1] FALSE
>> is.nan(NaN + NA_real_)
> [1] TRUE
>
> The R help files help(is.na) and help(is.nan) suggest that
> computations involving NA and NaN are indeterminate.
>
> It is faster to use the R convention; most operations are just
> handled by the hardware, without extra work.
>
> In cases like sum(x, na.rm=TRUE), the help file specifies that both NA
> and NaN are removed.
>
>
>
>
>>There is one NA but mulitple NaNs.
>>
>>And please re-read 'man memcmp': your cast is wrong.
>>
>>On 10/02/2014 06:52, Kevin Ushey wrote:
>>> Hi R-devel,
>>>
>>> I have a question about the differentiation between NA and NaN values
>>> as implemented in R. In arithmetic.c, we have
>>>
>>> int R_IsNA(double x)
>>> {
>>>  if (isnan(x)) {
>>> ieee_double y;
>>> y.value = x;
>>> return (y.word[lw] == 1954);
>>>  }
>>>  return 0;
>>> }
>>>
>>> ieee_double is just used for type punning so we can check the final
>>> bits and see if they're equal to 1954; if they are, x is NA, if
>>> they're not, x is NaN (as defined for R_IsNaN).
>>>
>>> My question is -- I can see a substantial increase in speed (on my
>>> computer, in certain cases) if I replace this check with
>>>
>>> int R_IsNA(double x)
>>> {
>>>  return memcmp(
>>>  (char*)(&x),
>>>  (char*)(&NA_REAL),
>>>  sizeof(double)
>>>  ) == 0;
>>> }
>>>
>>> IIUC, there is only one bit pattern used to enc

[Rd] Question re: NA, NaNs in R

[Kevin Ushey]

Hi R-devel,

I have a question about the differentiation between NA and NaN values
as implemented in R. In arithmetic.c, we have

int R_IsNA(double x)
{
if (isnan(x)) {
ieee_double y;
y.value = x;
return (y.word[lw] == 1954);
}
return 0;
}

ieee_double is just used for type punning so we can check the final
bits and see if they're equal to 1954; if they are, x is NA, if
they're not, x is NaN (as defined for R_IsNaN).

My question is -- I can see a substantial increase in speed (on my
computer, in certain cases) if I replace this check with

int R_IsNA(double x)
{
return memcmp(
(char*)(&x),
(char*)(&NA_REAL),
sizeof(double)
) == 0;
}

IIUC, there is only one bit pattern used to encode R NA values, so
this should be safe. But I would like to be sure:

Is there any guarantee that the different functions in R would return
NA as identical to the bit pattern defined for NA_REAL, for a given
architecture? Similarly for NaN value(s) and R_NaN?

My guess is that it is possible some functions used internally by R
might encode NaN values differently; ie, setting the lower word to a
value different than 1954 (hence being NaN, but potentially not
identical to R_NaN), or perhaps this is architecture-dependent.
However, NA should be one specific bit pattern (?). And, I wonder if
there is any guarantee that the different functions used in R would
return an NaN value as identical to R_NaN (which appears to be the
'IEEE NaN')?

(interested parties can see + run a simple benchmark from the gist at
https://gist.github.com/kevinushey/8911432)

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Format an empty data frame

[Kevin Ushey]

This behaviour is fixed in R-devel (tested with r64727)

> format( data.frame() )
data frame with 0 columns and 0 rows

-Kevin

On Thu, Jan 23, 2014 at 11:59 PM, Yihui Xie  wrote:
> Hi,
>
> Here seems to be a corner case in which format() fails:
>
>> format(data.frame())
> Error in .subset2(x, i, exact = exact) : subscript out of bounds
>
>> sessionInfo()
> R version 3.0.2 (2013-09-25)
> Platform: x86_64-pc-linux-gnu (64-bit)
>
> locale:
>  [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
> LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
>  [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
> LC_PAPER=en_US.UTF-8   LC_NAME=C
>  [9] LC_ADDRESS=C   LC_TELEPHONE=C
> LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods   base
>
>
> Regards,
> Yihui
> --
> Yihui Xie 
> Web: http://yihui.name
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Question about writing portable packages

[Kevin Ushey]

I think the key word here is _other_ packages. It's entirely okay to call
your package's own compiled code through the .Call interface (and the code
you write may link to other packages; the obvious example being Rcpp code
you write); however, it is not portable to use .Call to call compiled code
from another package directly. This is because package authors are not
required to maintain a consistent .Call interface to their compiled code;
rather, they just have to make sure the direct user-facing R exported
functions retain a stable interface over different versions (or, if
changed, are done so with prior warning etc.)

Ultimately, if your package passes 'R CMD check' with no errors, warnings
or notes, you should be okay for CRAN submission.

-Kevin


On Thu, Jun 6, 2013 at 6:54 AM,  wrote:

> Dear R-devel list,
>
> I am creating an R package that includes C++ code. I tried to install it
> both in Linux and Windows and it worked. I load the C code via the
> NAMESPACE file using useDynLib.
>
> Now I am revising once again whether everything is ready before
> submitting the package to CRAN and I have read in
> http://cran.r-project.org/doc/manuals/R-exts.html#Writing-portable-packages
> that “It is not portable to call compiled code in R or other packages
> via .Internal, .C, .Fortran, .Call or .External, since such interfaces
> are subject to change without notice and will probably result in your
> code terminating the R process”.
>
> However, the R function of my package that calls the C++ code uses the
> Call function.
>
> Does it mean that I cannot submit my package to CRAN because the
> portability is not guaranteed?. Then, how could I to call C++ code so
> that my package were portable?.
>
> I hope you can clarify my doubts. I have seen the collection of prior
> postings to the list, but I have not found a specifical answer to my
> doubts .
>
> Thank you very much in advance.
>
> Best regards,
>
> Guillermo
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>



-- 
-Kevin

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel