Re: [Rd] Big speedup in install.packages() by re-using connections

[Jeroen Ooms]

I'd like to raise this again now that 4.4 is out.

Below is a more complete patch which includes a function to properly
cleanup libcurl when R quits. Implementing this is a little tricky
because libcurl is a separate "module" in R, perhaps there is a better
way, but this works:

  view: https://github.com/r-devel/r-svn/pull/166/files
  patch: https://github.com/r-devel/r-svn/pull/166.diff

The old patch is still there as well, which is meant a minimal proof
of concept to test the performance gains for reusing the connection:

  view: https://github.com/r-devel/r-svn/pull/155/files
  patch: https://github.com/r-devel/r-svn/pull/155.diff

Performance gains are greatest on high-bandwidth servers when
downloading many files from the same server (e.g. packages from a cran
mirror). In such cases, currently over 90% of the total time is spent
on initiating and tearing town a separate SSL connection for each file
download.

Thoughts?



On Sat, Mar 2, 2024 at 3:07 PM Jeroen Ooms  wrote:
>
> Currently download.file() creates and terminates a new TLS connection
> for each download. This creates a lot of overhead which is expensive
> for both client and server (in particular the TLS handshake). Modern
> internet clients (including browsers) re-use connections for many http
> requests.
>
> We can do this in R by creating a persistent libcurl "multi-handle".
> The R libcurl implementation already uses a multi-handle, however it
> destroys it after each download, which defeats the purpose. The
> purpose of the multi-handle is to keep it alive and let libcurl
> maintain a persistent connection pool. This is particularly relevant
> for install.packages() which needs to download many files from one and
> the same server.
>
> Here is a bare minimal proof of concept patch that re-uses one and the
> same multi-handle for all requests in R:
> https://github.com/r-devel/r-svn/pull/155/files
>
> Some quick benchmarking shows that this can lead to big speedups for
> download.packages() on high-bandwidth servers (such as CI). This quick
> test to download 100 packages from CRAN showed more than 10x speedup
> for me: https://github.com/r-devel/r-svn/pull/155
>
> Moreover, I think this may make install.packages() more robust. In CI
> build logs that download many packages, I often see one or two
> downloads randomly failing with a TLS-connect error. I am hopeful this
> problem will disappear when using a single connection to the CRAN
> server to download all the packages.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Big speedup in install.packages() by re-using connections

[Jeroen Ooms]

Currently download.file() creates and terminates a new TLS connection
for each download. This creates a lot of overhead which is expensive
for both client and server (in particular the TLS handshake). Modern
internet clients (including browsers) re-use connections for many http
requests.

We can do this in R by creating a persistent libcurl "multi-handle".
The R libcurl implementation already uses a multi-handle, however it
destroys it after each download, which defeats the purpose. The
purpose of the multi-handle is to keep it alive and let libcurl
maintain a persistent connection pool. This is particularly relevant
for install.packages() which needs to download many files from one and
the same server.

Here is a bare minimal proof of concept patch that re-uses one and the
same multi-handle for all requests in R:
https://github.com/r-devel/r-svn/pull/155/files

Some quick benchmarking shows that this can lead to big speedups for
download.packages() on high-bandwidth servers (such as CI). This quick
test to download 100 packages from CRAN showed more than 10x speedup
for me: https://github.com/r-devel/r-svn/pull/155

Moreover, I think this may make install.packages() more robust. In CI
build logs that download many packages, I often see one or two
downloads randomly failing with a TLS-connect error. I am hopeful this
problem will disappear when using a single connection to the CRAN
server to download all the packages.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] CRAN checks for release of a package with new vignette engine

[Jeroen Ooms]

On Mon, Feb 26, 2024 at 5:50 PM Christophe Dervieux  wrote:
>
> Hi,
>
> I am trying to release a new version of the quarto R package. This new
> version is adding support for a new vignette engine that will use quarto
> CLI (https://quarto.org) when available. The vignettes inside the package
> itself are '.qmd' files built with quarto.
>
> While developing the feature, I noticed that R CMD check will query for
> vignette engines information, and it will find engine in already installed
> package (with `tools:::vignetteEngine`). So a package adding a new engine,
> and using this engine for its vignette only works when the package is
> installed before check.


This looks similar to https://bugs.r-project.org/show_bug.cgi?id=18191

Based on what I can see from the public cran incoming files [1,2], I
suspect the Debian check is loading the CRAN version of quarto (1.3),
rather than the one being checked (1.4), to test vignettes. As a
result it does not recognize the new qmd type vignettes introduced in
this version.

If this is indeed the case, the problem will resolve itself once the
submission is approved.

 [1] https://cran.r-project.org/incoming/archive/quarto_1.4.tar.gz
 [2] 
https://win-builder.r-project.org/incoming_pretest/quarto_1.4_20240221_175256/

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] new maintainer for CRAN package XML

[Jeroen Ooms]

On Mon, Jan 22, 2024 at 3:51 PM Uwe Ligges
 wrote:
>
> Dear package developers,
>
> the CRAN team (and Professor Ripley in particular) has been the defacto
> maintainer of CRAN package 'XML'.
> Our hope was that maintainers of packages depending on XML will migrate
> to other packages for reading XML structures. This has not happened and
> we still see dozens of strong dependencies on XML.

How is this hope communicated? Many R users assume that XML package is
in great shape and the preferable choice because it is maintained by
the CRAN team and r-core members.

Perhaps one could follow the precedent from the rgdal retirement, and
set a deadline.

One way to communicate this effectively would be by introducing a
formal deprecation field in the package description. This could then
be displayed on the XML CRAN html page, and when loading the package
interactively. Other packages that import such a deprecated package
could be given a CMD check warning.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Inquiry Regarding Package Organization in CRAN

[Jeroen Ooms]

On Fri, Jan 19, 2024 at 8:41 PM Protsak Andriy via R-package-devel
 wrote:
>
> Hi all!
>
>
>
> My name is Andriy, and I’m a student at University of Alcalá, currently 
> working on my final year project.
>
>
>
> I’m tasked with organizing the R packages developed by our university that 
> are currently available on CRAN. The goal is to enhance their 
> discoverability, to achieve this the initial focus is on exploring the 
> possibility of renaming the packages so that they share a common prefix, 
> making it easier for uses to locate them in the package list.
>
> If you believe there are alternative strategies to achieve a similar result, 
> please feel free to share your perspective.



You could also set up a cranlike package repository on r-universe,
where you include all the packages from your university (both those
from CRAN and others). This will provide a nice overview of all the
packages in your institute and makes them discoverable via web UI and
r-universe search engine.

Many other organisations are represented here as well, some examples:

 - https://ropensci.r-universe.dev
 - https://lcbc-uio.r-universe.dev
 - https://vimc.r-universe.dev
 - https://kwb-r.r-universe.dev
 - https://mrcieu.r-universe.dev
 - https://inbo.r-universe.dev

And many more from: https://r-universe.dev/organizations/. Getting
started is easy, see:
https://github.com/r-universe-org/help?tab=readme-ov-file

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] save.image Non-responsive to Interrupt

[Jeroen Ooms]

On Tue, May 2, 2023 at 3:29 PM Martin Maechler
 wrote:
>
> > Ivan Krylov
> > on Tue, 2 May 2023 14:59:36 +0300 writes:
>
> > В Sat, 29 Apr 2023 00:00:02 +
> > Dario Strbenac via R-devel  пишет:
>
> >> Could save.image() be redesigned so that it promptly responds to
> >> Ctrl+C? It prevents the command line from being used for a number of
> >> hours if the contents of the workspace are large.
>
> > This is ultimately caused by serialize() being non-interruptible. A
> > relatively simple way to hang an R session for a long-ish time would
> > therefore be:
>
> > f <- xzfile(nullfile(), 'a+b')
> > x <- rep(0, 1e9) # approx. 8 gigabytes, adjust for your RAM size
> > serialize(x, f)
> > close(f)
>
> > This means that calling R_CheckUserInterrupt() between saving
> > individual objects is not enough: R also needs to check for interrupts
> > while saving sufficiently long vectors.
>
> > Since the serialize() infrastructure is carefully written to avoid
> > resource leaks on allocation failures, it looks relatively safe to
> > liberally sprinkle R_CheckUserInterrupt() where it makes sense to do
> > so, i.e. once per WriteItem() (which calls itself recursively and
> > non-recursively) and once per every downstream for loop iteration.
> > Valgrind doesn't show any new leaks if I apply the patch, interrupt
> > serialize() and then exit. R also passes make check after the applied
> > patch.
>
> > Do these changes make sense, or am I overlooking some other problem?
>
> Thank you, Ivan!
>
> They do make sense... but :
>
> OTOH, in the past we have had to *disable*  R_CheckUserInterrupt()
> in parts of R's code because it was too expensive,
> {see current src/main/{seq.c,unique.c}  for a series of commented-out
>  R_CheckUserInterrupt() for such speed-loss reasons}
>
> so  adding these may need a lot of care when we simultaneously
> want to remain  efficient for "morally valid" use of serialization...
> where we really don't want to pay too much of a premium.

Alternatively, one could consider making R throttle or debounce calls
to R_CheckUserInterrupt such that a repeated calls within x time are
ignored, cf: https://www.freecodecamp.org/news/javascript-debounce-example/

The reasoning being that it may be difficult for (contributed) code to
determine when/where it is appropriate to check for interrupts, given
varying code paths and cpu speed. Maybe it makes more sense to call
R_CheckUserInterrupt frequently wherever it is safe to do so, and let
R decide if reasonable time has elapsed to actually run the (possibly
expensive) ui check again.

Basic example: https://github.com/r-devel/r-svn/pull/125/files




>
> {{ saving the whole user workspace is not "valid" in that sense
>in my view.  I tell all my (non-beginner) Rstudio-using
>students they should turn *off* the automatic saving and
>loading at session end / beginning; and for reproducibility
>only saveRDS() [or save()] *explicitly* a few precious
>objects ..
> }}
>
> Again, we don't want to punish people who know what they are
> doing, just because other R users manage to hang their R session
> by too little thinking ...
>
> Your patch adds 15 such interrupt checking calls which may
> really be too much -- I'm not claiming they are: with our
> recursive objects it's surely not very easy to determine the
> "minimally necessary" such calls.
>
> In addition, we may still consider adding an extra optional
> argument, say   `check.interrupt = TRUE`
> which we may default to TRUE when  save.image() is called
> but e.g., not when serialize() is called..
>
> Martin
>
> > --
> > Best regards,
> > Ivan
> > x[DELETED ATTACHMENT external: Rd_IvanKrylov_interrupt-serialize.patch, 
> text/x-patch]
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] Single page reference manual in html format

[Jeroen Ooms]

able standard, and I therefore cannot help.
>
> My advice to everyone involved in running CRAN, and I do solely from a 
> disabled person's perspective, is to make it hard to do the wrong thing, and 
> much easier to do the right things. I'm sure that we'll all find the results 
> are an improvement for everyone. After all, I could ask when you last chose 
> to use the ramp in preference to the steps to get into a building.
>
> Please consider the HTML alternative to any pdf, in any context, but 
> especially in anything relating to R.
>
> All the best,
> Jonathan
>
>
>
>
>
> -Original Message-
> From: R-package-devel  On Behalf Of 
> Deepayan Sarkar
> Sent: Saturday, 16 April 2022 6:36 am
> To: Duncan Murdoch 
> Cc: Jeroen Ooms ; R Package Development 
> 
> Subject: Re: [R-pkg-devel] Single page reference manual in html format
>
> On Fri, Apr 15, 2022 at 11:48 PM Duncan Murdoch  
> wrote:
> >
> > On 15/04/2022 1:16 p.m., Deepayan Sarkar wrote:
> > > On Sat, Apr 9, 2022 at 8:35 PM Jeroen Ooms  wrote:
> > >>
> > >> Is there a way to generate the package reference manual in a (one
> > >> page) html format? It would be nice to access reference manuals
> > >> from devices without a pdf reader.
> > >
> > > I don't think so, but it would certainly be a nice feature to have.
> > >
> > > Most of the necessary code is already there, so it shouldn't be too
> > > difficult. We'll work on it.
> >
> > I think this would be better done in a contributed package rather than
> > in core R, because there are lots of customizations that people would want:
> >
> > - Some people would want to execute the example code, like pkgdown does.
> > - Some people would want each help page in a separate HTML page (like
> > pkgdown), but with links to step through all the pages (I don't think
> > pkgdown does this).
> > - People would want to customize the look of the page, possibly with
> > syntax highlighting, or just different fonts and layouts.
> >
> > Adding an "all in one page" option to pkgdown might be the easiest way
> > to do this, because they already have a lot of customization
> > possibilities.  But there are probably other "make a web page for this
> > package" packages out there.
>
> Yes, that might be a good alternative.
>
> > If you do include it in base R, please at least allow the CSS to be
> > customized.
>
> Sure. So far I was only thinking of adding a couple of arguments to Rd2HTML()
>
> - 'standalone=TRUE' (FALSE would skip the header and footer), and possibly
>
> - 'Rhtml=FALSE' (TRUE would wrap the examples inside 
>
> This should make it easier to build up whatever is desired, regardless of 
> what base R implements. HTMLheader() already has an option to specify the CSS 
> file, so that could be used to create the header.
>
> Best,
> -Deepayan
>
> > Duncan Murdoch
>
> __
> R-package-devel@r-project.org mailing list
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-develdata=04%7C01%7Ca.j.godfrey%40massey.ac.nz%7Cff60d60fe13f4d00e28e08da1f0ef10b%7C388728e1bbd0437898dcf8682e644300%7C1%7C0%7C637856446305715235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=6e8ipmcWsUoviATnNU0KpPDzkozRKUOPMfqy2CtAgPg%3Dreserved=0
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] R, Rust and CRAN

[Jeroen Ooms]

On Sat, Nov 12, 2022 at 12:49 AM Simon Urbanek
 wrote:
>
> this does not directly address your question, but I think it would make a lot 
> of sense to standardize the process, given how many issues there were with 
> packages using Rust (mainly not detecting compilers correctly, not supplying 
> source code, unsolicited writing into user's directories, not checking 
> binaries etc.). Doing this right is not entirely trivial which is presumably 
> where the "friction" comes from.

All we need is the rustc/cargo toolchain to be installed on the CRAN
win/mac builders (it already is on Fedora/Debian). As mentioned above
(and before in e.g. https://jeroen.github.io/erum2018), rust uses the
native C FFI, does not need any runtime library, and can be called
directly from an R package using the C interface.  There is really no
need for frameworks or engines (like Rcpp/rJava/V8), this is precisely
what makes Rust so nice as an embedded language, and why it is being
adopted in many C projects such the Linux kernel, that would never
want to deal with Java/C++/JS.

> I'm not a Rust user myself, but just based on the other languages that have 
> interfaces from R I would think that Rust users could coalesce and write a 
> package that does the heavy lifting and can be used by packages that contain 
> Rust code as a foundation - that's what most other language interfaces do. 
> (It's quite possible that some of the projects you listed would be a good 
> start). I would not recommend putting that burden onto each package author as 
> it screams maintenance nightmare.

I understand the sentiment but Rust is very different from e.g. Java.
There really isn't much "heavy lifting" to do, because there are no
complex type conversions or runtime library involved. If the same
structs are used in C and Rust, the C functions can directly call Rust
functions and vice versa. Therefore it is possible for libraries to
incrementally port pieces of C code to Rust without breaking the ABI.
Just have a look at the hello world examples such as:
https://github.com/r-rust/hellorust or for a real world example:
https://cran.r-project.org/package=gifski

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] Unsigned binary

[Jeroen Ooms]

On Fri, Jun 17, 2022 at 2:46 PM Scott McAllister 
wrote:

> Thank you for your update on this
>
> If you can keep me updated on when version 4.2.1 is rereleased with the
> interim code signing certificate, that would be fantastic


Signed (unofficial) builds are also available from
https://github.com/r-windows/r-base/releases

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Package build errors with parallel make on Ubuntu 18.04

[Jeroen Ooms]

After upgrading to R-4.2 we are seeing package build errors on Ubuntu 18.04:
   make[1]: *** read jobs pipe: No such file or directory.
or:
   make[1]: *** write jobserver: Bad file descriptor.  Stop.

The problem seems to get triggered by the use of parallel make, but I
cannot find the underlying cause. It does not happen on ubuntu 20 or 22, so
I suspect the version of 'make' is related.

Attached a dockerfile with a minimal reproducible example (run with: docker
build .)
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[R-pkg-devel] Single page reference manual in html format

[Jeroen Ooms]

Is there a way to generate the package reference manual in a (one
page) html format? It would be nice to access reference manuals from
devices without a pdf reader.

We can use `R CMD Rd2pdf {pkgdir}` to generate the reference manual in
pdf format, but afaict `R CMD Rd2text` and `R CMD rdconv` only convert
single Rd files, and do not support generating a full package
reference manual.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[Rd] Update on rtools4 and ucrt support

[Jeroen Ooms]

Hi all,

I received some questions this week about rtools4 (the windows
compiler bundle) in particular regarding support for ucrt, so below a
brief summary of the status quo:

As of May 2021, rtools4 has full support for ucrt. The toolchain
configuration is based on the upstream msys2 configuration, which are
very stable, and widely used by other open source projects as well as
the mingw-w64 developers themselves. The latest builds of rtools4 now
contain 3 toolchains:

 - c:\rtools40\mingw32: the 32-bit gcc-8-3.0 toolchain used as of R 4.0.0
 - c:\rtools40\mingw64: the 64-bit gcc-8-3.0 toolchain used as of R 4.0.0
 - c:\rtools40\ucrt64: a new 64-bit gcc-10.3.0 toolchain targeting ucrt

The total install size is about 1gb. Hence, if R were to switch to
ucrt at some point, users and sysadmins that have installed rtools4
after May 2021 are already equipped with proper toolchains for
building packages for both R 4.0+ as well as a potential ucrt versions
of R.

As before, for each of these toolchains, all extra libraries needed by
CRAN packages can easily be installed in rtools4 through pacman [1].
All system libraries in rtools-packages [2] have ucrt64 binaries [3].
When users contribute an update or a new rtools package, the CI
automatically builds and checks binaries for each of the above
toolchains, e.g [4]. The process is 100% automatic, transparent, and
reproducible. This provides a degree of accountability, and makes it
easy for R package authors to suggest improvements for the C/C++
libraries that they depend on (many have done so in the past 2 years).

Rtools4 is preinstalled on major CI/cloud services such as GitHub
actions. Popular open-source projects such as Apache-Arrow and TileDB
are already using the rtools4 toolchains to automatically build and
test their C++ libraries, as well as R bindings, for each commit, on
all target architectures (including ucrt64). Any R package author can
use the the same free services to check their packages on all compile
targets using rtools4 toolchains [5]. The r-devel CI tool on
https://r-devel.github.io checks every commit to base-R using ucrt64
toolchain from rtools4, which has proven to be very stable.

I am also aware that Tomas Kalibera also provides alternative
"experimental ucrt toolchain": a 6gb tarball with manually built
things on his personal machine. It is unclear to me why it was decided
to take this approach; it is certainly not needed to support ucrt
(ucrt is literally one flag in the toolchain configuration).
Fortunately, the ucrt tooclchains from rtools4 and Tomas Kalibera use
the same version of mingw-w64 and gcc, and are fully compatible, so
package authors could still use the rtools4 ucrt compilers icw the
R-devel-ucrt version that was built using this experimental toolchain
[5].

We spent an enormous amount of effort in the past years standardising
the Windows build tooling, and making the infrastructure automated,
open and accessible, such that everyone can learn how this works and
get involved. Many people have. Today if you install R and Rtools4 on
Windows, things "just work", regardless of whether this is a student
laptop, university server, or online CI system. Anyone can build R
packages, base-R, or any of the system libraries, following the steps,
and using standard tooling that other open source projects use. I
think it would be a big step back of R-core decides to go back to a
black-box system that is so opaque and complex that only one person
knows how it works, and would make it much more difficult for
students, universities, and other organisations to build R packages
and libraries on Windows.

Jeroen


[1] https://github.com/r-windows/docs/blob/master/rtools40.md#readme
[2] https://github.com/r-windows/rtools-packages
[3] https://cran.r-project.org/bin/windows/Rtools/4.0/
[4] https://github.com/r-windows/rtools-packages/pull/221
[5] https://github.com/r-windows/docs/blob/master/ucrt.md

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Feature request: Change default library path on Windows

[Jeroen Ooms]

On Sun, Jul 25, 2021 at 12:15 AM Steve Haroz  wrote:
>
> Hello,
>
> I'd like to propose moving the default library install location on Windows 
> from:
> %USERPROFILE%/Documents/R
> to some other location such as:
> %USERPROFILE%/R
>
> For many users the Documents folder is backed up or synchronized.
> Installing libraries thrashes Documents, and it causes synchronization
> issues with Dropbox (I confirm this one), OneDrive, and users with
> Network IT policies.
>
> The vast majority of R users won't touch that folder and don't need it
> backed up. And, its contents are not really "documents".

I very much support this request. R's behavior of defaulting HOME to
an unusual (legacy) directory is also a source of bugs and
interoperability problems:

R packages that interface c/c++ libraries that require a user config
file (libssh, libgit2, etc) sometimes cannot find the user config
because it gets stored on a different location than expected. A good
example is 'git'. The 'git for windows' command line utility stores my
user config in C:\users\Jeroen\.gitconfig. However, for R packages
that link to libgit2 (e.g. git2r, gert) the user config gets
loaded/stored from C:\users\Jeroen\Documents\.gitconfig because
libgit2 inherits the R home directory. Having 2 different git
configurations is obviously confusing for users.

Another problem is that when you start R from "windows bash" or the
msys2 shell (including "git for windows" or rtools40), the "~"
directory is different than in RGui. This is because these shells do set
the HOME variable (to the usual value), and R inherits that.

For example: when I start R in RGui for Windows I see:

  > normalizePath("~")
  [1] "C:\\Users\\jeroen\\Documents"

But when I start R or Rterm from the "git for windows" shell we get:

  > normalizePath("~")
  [1] "C:\\Users\\jeroen"

An ambiguous home directory is obviously going to break R code that
refers to paths in your home.

Because these problems affect all Windows users, I think the benefits
of fixing the default behavior easily outweigh the
backward-compatibility and "there is a documented workaround"
arguments.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Patch: port getbyhostname to getaddrinfo

[Jeroen Ooms]

Perhaps now that nanohttp has been removed, it is a good time to do
some related cleanups.

In both Linux and Windows, gethostbyname and gethostbyaddr are marked
as obsolete. POSIX.1-2008 removes the specifications of
gethostbyname(), gethostbyaddr(), and h_errno, recommending the use of
getaddrinfo(3) and getnameinfo(3) instead.

Webview: https://github.com/r-devel/r-svn/pull/42/files
Patch: https://github.com/r-devel/r-svn/pull/42.diff

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] problem adding gdb to RTOOLS40 on Windows

[Jeroen Ooms]

On Thu, Apr 15, 2021 at 10:19 AM Bravington, Mark (Data61, Hobart)
 wrote:
>
> Excellent, thank you all. Doing "pacman -Syu" before "pacman -S gdb" did 
> the trick, and yes it's a different version of gdb compared to what pacman 
> was previously trying to install. Might be worth updating the RTOOLS40 
> documentation to recommend doing -Syu first.

OK I have added a line to the docs. FYI, you can combine the two
commands into one:

  pacman -Sy mingw-w64-x86_64-gdb

This will first update the repository index (-y) and the install the package.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Unicode characters in ISO8859-15 locale

[Jeroen Ooms]

The hunspell package uses the code below to replace curly quotes (aka
fancyquotes) with a regular ascii quotes that are needed for check
spelling:

chartr("\u2019", "'", input)

As of last week this stopped working on CRAN in the Linux server that
runs in ISO8859-15 locale. From the error message, it seems that R no
longer parses the escaped unique string, which gets turned into
"".

Is this expected?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] installling R-devel on Windows

[Jeroen Ooms]

On Sat, Dec 5, 2020 at 3:00 PM Gabor Grothendieck
 wrote:
>
> When I try to install r-devel on Windows all I get is this.  No other
> files.  This also occurred yesterday as well.

It just tested it to be sure, but it works fine for me. Are you using
the official installer from
https://cran.r-project.org/bin/windows/base/rdevel.html ?

The default install path is not R-test C:\Program Files\R\R-devel.
Perhaps you have old files lingering from previous installations that
cause permission problems during the installation process?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Operations with long altrep vectors cause segfaults on Windows

[Jeroen Ooms]

On Tue, Sep 8, 2020 at 11:44 PM Jeroen Ooms  wrote:
>
> On Tue, Sep 8, 2020 at 5:20 PM Tomas Kalibera  
> wrote:
> >
> > On 9/8/20 4:48 PM, Hugh Parsonage wrote:
> > > Unfortunately I only get
> > >
> > > [Thread 21752.0x4aa8 exited with code 3221225477]
> > > [Thread 21752.0x4514 exited with code 3221225477]
> > > [Thread 21752.0x3f10 exited with code 3221225477]
> > > [Inferior 1 (process 21752) exited with code 0305]
> > >
> > > (I'm guessing I would need to build an instrumented version of R, or
> > > can R be debugged using gdb with an off-the-shelf installation?)
> >
> > No, the default build lacks debug symbols. You need a build with debug
> > symbols, and if you can reproduce in a build without compiler
> > optimizations (-O0), the backtrace may be easier to interpret. Some bugs
> > however "disappear" when optimizations are disabled. You can build R
> > from source (and there may be debug builds provided by someone else
> > (Jeroen?)).
>
> Debug builds for each revision are available from
> https://r-devel.github.io . To download the installer you need to
> click the github icon in the last column in the table. You need to be
> signed in with a (free) Github account in order to download builds
> (artifacts) from Github actions. It will show download links for both
> the regular installer and installer with debug symbols.
>
> In other news, the https://r-devel.github.io table also shows that the
> fix that martin committed is segfaulting on 32-bit.

Sorry that was inaccurate, it is not segfaulting at all, but the unit
test is raising an error on 32-bit.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Operations with long altrep vectors cause segfaults on Windows

[Jeroen Ooms]

On Tue, Sep 8, 2020 at 5:20 PM Tomas Kalibera  wrote:
>
> On 9/8/20 4:48 PM, Hugh Parsonage wrote:
> > Unfortunately I only get
> >
> > [Thread 21752.0x4aa8 exited with code 3221225477]
> > [Thread 21752.0x4514 exited with code 3221225477]
> > [Thread 21752.0x3f10 exited with code 3221225477]
> > [Inferior 1 (process 21752) exited with code 0305]
> >
> > (I'm guessing I would need to build an instrumented version of R, or
> > can R be debugged using gdb with an off-the-shelf installation?)
>
> No, the default build lacks debug symbols. You need a build with debug
> symbols, and if you can reproduce in a build without compiler
> optimizations (-O0), the backtrace may be easier to interpret. Some bugs
> however "disappear" when optimizations are disabled. You can build R
> from source (and there may be debug builds provided by someone else
> (Jeroen?)).

Debug builds for each revision are available from
https://r-devel.github.io . To download the installer you need to
click the github icon in the last column in the table. You need to be
signed in with a (free) Github account in order to download builds
(artifacts) from Github actions. It will show download links for both
the regular installer and installer with debug symbols.

In other news, the https://r-devel.github.io table also shows that the
fix that martin committed is segfaulting on 32-bit.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R 4.0.2 64-bit Windows hangs

[Jeroen Ooms]

On Wed, Aug 26, 2020 at 7:54 PM Tomas Kalibera  wrote:
>
> On 8/25/20 6:14 PM, Tomas Kalibera wrote:
> > On 8/22/20 9:33 PM, Jeroen Ooms wrote:
> >> On Sat, Aug 22, 2020 at 9:10 PM Tomas Kalibera
> >>  wrote:
> >>> On 8/22/20 8:26 PM, Tomas Kalibera wrote:
> >>>> On 8/22/20 7:58 PM, Jeroen Ooms wrote:
> >>>>> On Sat, Aug 22, 2020 at 8:39 AM Tomas Kalibera
> >>>>>  wrote:
> >>>>>> On 8/21/20 11:45 PM, m19tdn+9alxwj7d2bmk--- via R-devel wrote:
> >>>>>>> Ah yes, this is related. I reported v2010 below, but it looks like
> >>>>>>> I was updated to this Insider Build overnight without my knowledge,
> >>>>>>> and conflated it with the new installation R v4 this morning.
> >>>>>>>
> >>>>>>> I will continue to look into the issue with the methods Tomas
> >>>>>>> mentioned.
> >>>>>> It is interesting that a rare 5 years old problem would re-appear on
> >>>>>> current Insider builds. Which build of Windows are you running
> >>>>>> exactly?
> >>>>>> I've seen another report about a crash on 20190.1000. It'd be
> >>>>>> nice to
> >>>>>> know if it is present also in newer builds, i.e. in 20197.
> >>>>> I installed the latest 20197 build in a vm, and I can indeed
> >>>>> reproduce
> >>>>> this problem.
> >>>>>
> >>>>> What seems to be happening is that R triggers an infinite
> >>>>> recursion in
> >>>>> Windows unwinding mechanism, and eventually dies with a stack
> >>>>> overflow. Attached a backtrace of the initial 100 frames of the main
> >>>>> thread (the pattern in the top ~30 frames continues forever).
> >>>>>
> >>>>> The microsoft blog doesn't mention anything related to exception
> >>>>> handling has changed in recent versions:
> >>>>> https://docs.microsoft.com/en-us/windows-insider/at-home/active-dev-branch
> >>>>>
> >>>>>
> >>>> Thanks, unfortunately that does not ring any bells (except below), I
> >>>> can't guess from this what is the underlying cause of the problem.
> >>>> There may be something wrong in how we use setjmp/longjmp or how
> >>>> setjmp/longjmp works on Windows.
> >>>>
> >>>> It reminds me of a problem I've been debugging few days ago, when
> >>>> longjump implementation segfaults on Windows 10 (recent but not
> >>>> Insider build) probably soon after unwinding the stack, but only with
> >>>> GCC 10 / MinGW 7 and only in one of the no-segfault tests, and only
> >>>> with -03 (not -O2, not with with -O3 -fno-split-loops). The problem
> >>>> was sensitive to these optimization options interestingly on the call
> >>>> site of long jump (do_abs), even when it was not an immediate caller
> >>>> of the longjump. I've not tracked this down yet, it will require
> >>>> looking at the assembly level, and I was suspecting a compiler error
> >>>> causing the compiler to generate code that messes with the stack or
> >>>> registers in a way that impacts the upcoming jump. But now as we have
> >>>> this other problem with setjmp/logjmp, the compiler may not be the top
> >>>> suspect anymore.
> >>>>
> >>>> I may not be able to work on this in the next few days or a week, so
> >>>> if anyone gets there first, please let me know what you find out.
> >>> Btw could you please try out if the UCRT build of R crashes as well in
> >>> the Insider Windows build ?
> >> Yes, it hangs in exactly the same way, except that the backtrace shows
> >>
> >>   ucrtbase!.intrinsic_setjmpex () from C:\WINDOWS\System32\ucrtbase.dll
> >>
> >> Instead of msvcrt!_setjmpex (as expected of course).
> >
> > Thanks. I found what is causing the problem I observed with
> > GCC10/stock Windows 10, I expect this is the same one as in the
> > Insider build.
> > I will investigate further,
> >
> > Tomas
> >
> It seems the problem is between MinGW-W64 and Windows, and really it
> causes both the reported crashes in an Insider build (I tested in 20197)
> and in my GCC 10 builds in a single "no-segfault" test. setjmp is
> implemented using Windows call _setjmpex, which has a second argument
> argument, which is set differently by MinGW based on GCC version. When I
> set this argument as MinGW-W64 did on early versions of GCC,
> mingw_getsp(), it fixes/hides the problems on my systems. Perl5 uses a
> similar workaround, but otherwise there is no solid base (documentation,
> specification, etc) I am aware of for this change, so this may take some
> more time to be properly fixed. Still, if anyone experiments with this
> workaround and finds a problem, please let me know. In particular, I am
> curious whether it works on earlier versions of Windows (at least with
> check-all, including recommended packages).

FYI, the problem has disappeared on Windows dev built 20201 (released
yesterday), so it may have been a Windows bug. That is not to say
there is no bug on the R/mingw side, but at least the current and past
releases of R are working again on the latest versions of Windows,
which is a big relief.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R 4.0.2 64-bit Windows hangs

[Jeroen Ooms]

On Sat, Aug 22, 2020 at 9:10 PM Tomas Kalibera  wrote:
>
> On 8/22/20 8:26 PM, Tomas Kalibera wrote:
> > On 8/22/20 7:58 PM, Jeroen Ooms wrote:
> >> On Sat, Aug 22, 2020 at 8:39 AM Tomas Kalibera
> >>  wrote:
> >>> On 8/21/20 11:45 PM, m19tdn+9alxwj7d2bmk--- via R-devel wrote:
> >>>> Ah yes, this is related. I reported v2010 below, but it looks like
> >>>> I was updated to this Insider Build overnight without my knowledge,
> >>>> and conflated it with the new installation R v4 this morning.
> >>>>
> >>>> I will continue to look into the issue with the methods Tomas
> >>>> mentioned.
> >>> It is interesting that a rare 5 years old problem would re-appear on
> >>> current Insider builds. Which build of Windows are you running exactly?
> >>> I've seen another report about a crash on 20190.1000. It'd be nice to
> >>> know if it is present also in newer builds, i.e. in 20197.
> >> I installed the latest 20197 build in a vm, and I can indeed reproduce
> >> this problem.
> >>
> >> What seems to be happening is that R triggers an infinite recursion in
> >> Windows unwinding mechanism, and eventually dies with a stack
> >> overflow. Attached a backtrace of the initial 100 frames of the main
> >> thread (the pattern in the top ~30 frames continues forever).
> >>
> >> The microsoft blog doesn't mention anything related to exception
> >> handling has changed in recent versions:
> >> https://docs.microsoft.com/en-us/windows-insider/at-home/active-dev-branch
> >>
> >
> > Thanks, unfortunately that does not ring any bells (except below), I
> > can't guess from this what is the underlying cause of the problem.
> > There may be something wrong in how we use setjmp/longjmp or how
> > setjmp/longjmp works on Windows.
> >
> > It reminds me of a problem I've been debugging few days ago, when
> > longjump implementation segfaults on Windows 10 (recent but not
> > Insider build) probably soon after unwinding the stack, but only with
> > GCC 10 / MinGW 7 and only in one of the no-segfault tests, and only
> > with -03 (not -O2, not with with -O3 -fno-split-loops). The problem
> > was sensitive to these optimization options interestingly on the call
> > site of long jump (do_abs), even when it was not an immediate caller
> > of the longjump. I've not tracked this down yet, it will require
> > looking at the assembly level, and I was suspecting a compiler error
> > causing the compiler to generate code that messes with the stack or
> > registers in a way that impacts the upcoming jump. But now as we have
> > this other problem with setjmp/logjmp, the compiler may not be the top
> > suspect anymore.
> >
> > I may not be able to work on this in the next few days or a week, so
> > if anyone gets there first, please let me know what you find out.
>
> Btw could you please try out if the UCRT build of R crashes as well in
> the Insider Windows build ?

Yes, it hangs in exactly the same way, except that the backtrace shows

 ucrtbase!.intrinsic_setjmpex () from C:\WINDOWS\System32\ucrtbase.dll

Instead of msvcrt!_setjmpex (as expected of course).

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R 4.0.2 64-bit Windows hangs

[Jeroen Ooms]

On Sat, Aug 22, 2020 at 8:39 AM Tomas Kalibera  wrote:
>
> On 8/21/20 11:45 PM, m19tdn+9alxwj7d2bmk--- via R-devel wrote:
> > Ah yes, this is related. I reported v2010 below, but it looks like I was 
> > updated to this Insider Build overnight without my knowledge, and conflated 
> > it with the new installation R v4 this morning.
> >
> > I will continue to look into the issue with the methods Tomas mentioned.
>
> It is interesting that a rare 5 years old problem would re-appear on
> current Insider builds. Which build of Windows are you running exactly?
> I've seen another report about a crash on 20190.1000. It'd be nice to
> know if it is present also in newer builds, i.e. in 20197.

I installed the latest 20197 build in a vm, and I can indeed reproduce
this problem.

What seems to be happening is that R triggers an infinite recursion in
Windows unwinding mechanism, and eventually dies with a stack
overflow. Attached a backtrace of the initial 100 frames of the main
thread (the pattern in the top ~30 frames continues forever).

The microsoft blog doesn't mention anything related to exception
handling has changed in recent versions:
https://docs.microsoft.com/en-us/windows-insider/at-home/active-dev-branch
#39431 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39432 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39433 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39434 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39435 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39436 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39437 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39438 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39439 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39440 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39441 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39442 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39443 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39444 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39445 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39446 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39447 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39448 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39449 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39450 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39451 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39452 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39453 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39454 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39455 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39456 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39457 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39458 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39459 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39460 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39461 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39462 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39463 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39464 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39465 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39466 0x7ffe105a8d77 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39467 0x7ffe105a8b07 in ntdll!RtlVirtualUnwind () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39468 0x7ffe105a7258 in ntdll!RtlInitializeExtendedContext () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39469 0x7ffe10639d5e in ntdll!KiUserExceptionDispatcher () from 
C:\WINDOWS\SYSTEM32\ntdll.dll
#39470 0x7ffe105a9584 in ntdll!RtlVirtualUnwind () 

Re: [Rd] Experimental CI tool for R

[Jeroen Ooms]

On Thu, Jul 23, 2020 at 11:57 PM Simon Urbanek
 wrote:
>
> This is great! It is definitely a good basis to build on.
> I wonder why your macOS setup is so extremely stripped down (not even Cairo, 
> tcltk nor X11 - and not TeX, either) and as far from what we actually use as 
> possible (using gcc instead of clang, openblas etc.).
> How do you plan to go about managing the build flavors? I think it would be 
> great if there was a process whereby the builds could be updated so they are 
> more realistic and thus more helpful, but since the repo is completely 
> anonymous, it's unclear how one would go about that nor how it would be 
> governed (and where to put documentation).

Thanks for having a look at this.

Build scripts for GitHub actions are always stored in the workflows
directory in the same repository. The build-svn.yaml file contains the
commands used to prepare the server and build R on each of the
platforms. Here you can easily enable/disable features, or add another
flavor. In the same way you can test patches, you can use pull
requests to suggest changes to the build matrix. I have also added a
note about this in the readme.

On MacOS currently indeed we test a minimal configuration which
matches homebrew:
https://github.com/homebrew/homebrew-core/blob/master/Formula/r.rb#L35-L47
. The main reason is to minimize random build failures that we were
getting when downloading xquartz and mactex during the build process
(these are not preinstalled on the GHA builders, anc MacTex).

It would be great if you can help with adding a flavor to build a
cran-like MacOS installer.


> For obvious reasons the Windows one is the only complete one, but given the 
> requests for Homebrew-based package testing (independent of CRAN) it would be 
> useful to publish the artefacts as well so that they could be used by GH 
> action workflows for packages. Cleary we could just fork it, but I guess it 
> would make more sense if this was a coordinated effort.

Of course, 100% agree this should be a coordinated effort.  Ideally we
hope some modern tooling can be adopted upstream, as for most other
open source projects, where CI is a standard part of the development
process, such that cross-platform building and testing is automated
and transparent.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] configure failed with curl 7.71.1

[Jeroen Ooms]

On Sat, Jul 25, 2020 at 10:12 AM Marco Atzeri  wrote:
>
> Hi dev,
>
> can someone confirm if it is a general R 4.0.4 problem
> or it is happening only on cygwin ?
>
> checking for curl/curl.h... yes
> checking if libcurl is version 7 and >= 7.28.0...
> configure: error: libcurl >= 7.28.0 library and headers are required
> with support for https
> *** ERROR: configure failed
>
> but https is available on curl

You should inspect the config.log or config.status file to see why the
version check is failing.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Experimental CI tool for R

[Jeroen Ooms]

Based on ideas from the R-core discussion panel at useR2020, I created
a little CI tool to make it easier to follow changes in R-devel, and
to write/test patches for R.

The tool is based on a Github mirror of the SVN, where each new commit
triggers a full make-check on 8 different system configurations. The
results are published on: https://r-devel.github.io which gives an
overview of the most recent revisions, including links to the build
logs, and a link to the (unsigned) Windows installer. As of yesterday,
it should be possible to inspect the build logs without signing in to
GitHub.

The system can also be used to develop and test patches for base-R.
Anyone can send pull-requests, which will trigger the same set of
builds. The check results and link to Windows installer will appear
under your pull request. Finally, GitHub makes it very easy to export
a pull request as a patch file, which is the format that R-core
members still like to use. More instructions are available on:
https://github.com/r-devel/r-svn#readme

I hope this tool can make cross-platform testing and contributing of
base-R slightly less painful, while we are still on SVN.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Changes in MiKTeX

[Jeroen Ooms]

MiKTeX released a major new version this week, with some breaking
changes that may be important for Windows users and sysadmins.

The MiKTeX versioning has changed to a date-based scheme. The previous
version was called 2.9, the distribution is now called miktex-20.6.

The default installation path has changed from "C:/Program
Files/MiKTeX 2.9/" to "C:/Program Files/MiKTeX/". Hence this may
require updating scripts that set the PATH to pdflatex and others. I
have updated the r-base build script, so people that use this to build
R locally need to make sure MikTeX is available at the new path.

Finally, the 20.6 release contains a bad bug that causes pdftex to not
work at all, including failing to build R manuals:
https://github.com/MiKTeX/miktex/issues/568 . It seems to have been
fixed and a new pdftex is rolled out via the miktex package manager,
so make sure to update the miktex packages after installing
miktex-20.6 (or wait for the 20.7 release).

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: use of the tcltk package crashes R 4.0.1 for Windows

[Jeroen Ooms]

On Mon, Jun 8, 2020 at 12:03 AM  wrote:
>
> I've committed the change to use Free instead of free in tcltk.c and
> sys-std.c (r78652 for R-devel, r78653 for R-patched).

Thank you! I can confirm that the example from above no longer crashes
in R--patched.

John, can you confirm that everything seems to work now in Rcmd with
today's R-patched build from CRAN?
https://cran.r-project.org/bin/windows/base/rpatched.html

Hopefully Peter will be able to tag a 4.0.2 hotfix release based on
4.0.1 + above patch, without going through the full release
procedure...

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: use of the tcltk package crashes R 4.0.1 for Windows

[Jeroen Ooms]

On Sun, Jun 7, 2020 at 5:53 PM  wrote:
>
> On Sun, 7 Jun 2020, peter dalgaard wrote:
>
> > So this wasn't tested for a month?
> >
> > Anyways, Free() is just free() with a check that we're not freeing a null 
> > pointer, followed by setting the pointer to NULL. At that point of tcltk.c, 
> > we have
> >
> >   for (objc = i = 0; i < length(avec); i++){
> >const char *s;
> >char *tmp;
> >if (!isNull(nm) && strlen(s = translateChar(STRING_ELT(nm, i{
> >//  tmp = calloc(strlen(s)+2, sizeof(char));
> >tmp = Calloc(strlen(s)+2, char);
> >*tmp = '-';
> >strcpy(tmp+1, s);
> >objv[objc++] = Tcl_NewStringObj(tmp, -1);
> >free(tmp);
> >}
> >if (!isNull(t = VECTOR_ELT(avec, i)))
> >objv[objc++] = (Tcl_Obj *) R_ExternalPtrAddr(t);
> >}
> >
> > and I can't see how tmp can be NULL at the free(), nor can I see it 
> > mattering if it is not set to NULL (notice that it goes out of scope with 
> > the for loop).
>
> Right. And the calloc->Calloc change doesn't look like an issue either
> -- just checking for a NULL.
>
> If the crash is happening in free() then that most likely means
> corrupted malloc data structures. Unfortunately that could be
> happening anywhere.

Writing R extensions, section 6.1.2 says: "Do not assume that memory
allocated by Calloc/Realloc comes from the same pool as used by
malloc: in particular do not use free or strdup with it."

I think the reason is that R uses dlmalloc for Calloc on Windows:
https://github.com/wch/r-source/blob/c634fec5214e73747b44d7c0e6f047fefe44667d/src/main/memory.c#L94-L103

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] use of the tcltk package crashes R 4.0.1 for Windows

[Jeroen Ooms]

On Sun, Jun 7, 2020 at 3:13 AM Fox, John  wrote:
>
> Hi,
>
> The following code, from the examples in ?TkWidgets , immediately crashes R 
> 4.0.1 for Windows:
>
> - snip 
> library("tcltk")
> tt <- tktoplevel()
> label.widget <- tklabel(tt, text = "Hello, World!")
> button.widget <- tkbutton(tt, text = "Push",
>  command = function()cat("OW!\n"))
> tkpack(label.widget, button.widget) # geometry manager
> - snip 


I can reproduce this. The backtrace shows the crash happens in
dotTclObjv  [/src/library/tcltk/src/tcltk.c@243 ]. This looks like a
bug that was introduced by commit 78408/78409 about a month ago. I
think the problem is that this commit changes 'calloc' to 'Calloc'
without changing the corresponding 'free' to 'Free'.

This has nothing to do with the Windows build or installation. Nothing
has changed in the windows build procedure between 4.0.0 and 4.0.1.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Debugging packages with compiled C code on Windows

[Jeroen Ooms]

On Mon, Jun 1, 2020 at 11:36 PM Sue McDonald  wrote:
>
> I have several related questions.
>
> 1.  Is it possible to use a GUI: Rstudio/Eclipse/Visual-studio to debug
> compiled code on Windows? Things that work on Eclipse for Windows do not
> work on Eclipse for Windows.
> 2.  R CMD INSTALL seems to override default attempts to provide
> CFLAGS="-DDEBUG -g3 -O0"
> 3.  Is it necessary to compile R with debug turned on?   One of the FAQs
> mentioned to compile R with make DEBUG=T.
> 4.  Using Rtools 4.0 and Jeroen's scripts for building R works great (many
> thanks). But does not seem to have an impact on optimization, other than
> including -gwarf-2.  It adds -DNDEBUG flag.  Is that sufficient for
> debugging compiled code in a package?  Obviously, I just need to debug
> package code, so does it matter?

If you use the official R-for-Windows installation, you should install
your R package with R CMD INSTALL --debug in order to keep debug
symbols. That is usually sufficient to get a backtrace with e.g
drmingw. See FAQ 8.4 of
https://cran.r-project.org/bin/windows/base/rw-FAQ.html

Coincidentally I am also working on a special debugging build of R,
which uses -g -O0 for all compiled code. It is not final yet, but it
should work. You can test it here:
https://github.com/r-windows/rtools-packages/pull/119

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] r-project.org SSL certificate issues

[Jeroen Ooms]

On Sat, May 30, 2020 at 11:40 PM Duncan Murdoch
 wrote:
>
> On 30/05/2020 5:23 p.m., Bob Rudis wrote:
> > I've updated the dashboard (https://rud.is/r-project-cert-status/)
> > script and my notifier script to account for the entire chain in each
> > cert.
>
> You never posted which certificate has expired.  Your dashboard shows
> they're all valid, but the download still fails, presumably because
> something not shown has expired.

To see the problem in R:

   certs <- openssl::download_ssl_cert('cran.r-project.org')
   as.list(certs[[3]])

Shows the root cert expires today.

> Hopefully someone who can actually act on this can figure out what needs
> doing.

The apache server will have a config entry SSLCertificateFile which
points to a cert bundle (in nginx servers this is called
"ssl_certificate"). If you open this in a text editor it contains the
3 certs, in PEM format, so 3 entires like this:

-BEGIN CERTIFICATE-
[base64 cert]
-END CERTIFICATE-

What you need to do is replace the final certificate with this one
(just copy-paste the base64 cert): https://crt.sh/?d=1720081 .Then
restart the server.

See here for details:
https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020
. This site talks about "For business processes that depend on very
old systems" but the reality is that this affects everything that
uses openssl for https, including curl, svn, etc.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Build failing on win64

[Jeroen Ooms]

As of commit 78536 earlier this morning the build is failing on
windows 64, see https://r-devel.github.io

I cannot immediately spot what is the problem. The build fails with:

installing 'sysdata.rda'
  make[3]: *** [../../../share/make/basepkg.mk:151: sysdata] Error 127
  make[2]: *** [Makefile.win:22: all] Error 2
  make[1]: *** [Makefile.win:32: R] Error 1

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] Is there an Rtools40 changelog?

[Jeroen Ooms]

On Sun, May 10, 2020 at 11:31 PM James Lamb  wrote:
>
> Hello,
>
> I am a maintainer on the LightGBM project, focused on that project's R
> package. The R package is not available on CRAN yet (we are working on it),
> so for now our users must build it from source.
>
> The package includes compilation of a C++ library, and we link to R.dll /
> R.so to use R-provided functions like Rprintf.
>
> With the release of R4.0 and Rtools40, we recently received reports from
> our users that they are unable to build our package on Windows systems with
> R 4.0 and Rtools 4.0 (https://github.com/microsoft/LightGBM/issues/3064).
>
> After some investigation, I've learned that the following changes in
> Rtools40 (relative to Rtools35) broke our installation process:
>
>- *gendef.exe* was removed
>- *mingw32-make.exe* was removed
>- paths like "mingw_64/bin" were changed to "mingw64/bin"
>

Some utilities that were previously bundled with all rtools
installations can now be installed with the package manager.

To install gendef use:

 pacman -S mingw-w64-{i686,x86_64}-tools

To install mingw32-make.exe use:

  pacman -S mingw-w64-{i686,x86_64}-make

To install both of the at once you can use:

  pacman -S mingw-w64-{i686,x86_64}-{tools,make}

Note that it's often better to use 'make.exe' which is included with
all rtools installations instead of mingw32-make.exe.

These changes were a result of switching to an msys2 based toolchain.
The default rtools40 installer only includes the things needed to
build CRAN packages or base-R. Extra stuff such as system libraries,
debuggers, cmake, etc, are all optionally available via the package
manager.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] Tcl.Tk Tktable package missing from R 4.0.0 on Windows

[Jeroen Ooms]

On Sun, May 3, 2020 at 6:15 PM Fox, John  wrote:
>
> Dear R-devel list members,
>
> The Tktable package for Tcl/Tk is apparently missing from the Windows 
> distribution of R 4.0.0. I (actually a user of the Rcmdr package) discovered 
> this when trying to use the new-data-set dialog in the Rcmdr package, 
> producing the error, "Tcl package 'Tktable' must be installed first."
>
> I believe the Tktable has been part of the R distribution for Windows since R 
> version 2.9.0, and is still present in the macOS distribution of R 4.0.0.
>
> I apologize for not discovering this problem prior to the release of R 4.0.0. 
> I did test the Rcmdr package under R 4.0.0 on both Windows and macOS, but not 
> every menu item and dialog on each platform.
>
> Does anyone have more information about this problem?

Sorry this was my mistake. I somehow missed tktable when updating the
tcltk bundle for rtools40. This didn't show up in any package check
either.

I've added it back now, it should work again in today's builds r-devel
and r-patched. Thanks for catching this.

Jeroen

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] "not a valid win32 application" with rtools40-x86_65.exe on Windows 10

[Jeroen Ooms]

On Thu, Apr 30, 2020 at 6:38 AM Spencer Graves
 wrote:
>
> Hello, All:
>
>
>"00install.out" from "R CMD check Ecfun_0.2-4.tar.gz" includes:
>
>
> Error:  package or namespace load failed for 'Ecfun':
>   .onLoad failed in loadNamespace() for 'rJava', details
>call: inDL(x, as.logical(local), as.logical(now), ...)
>error:  unable to load shared object 'c:/Program
> Files/R/R-4.0.0/library/rJava/libs/i386/rJava.dll':
>LoadLibrary failure: ^1 is not a valid win32 application
>

This is an error in loading the rJava package, so it is not related to
rtools40, and probably inappropriate for this mailing list.

As Simon suggested, you may have to install the 32-bit Java JDK. See
also this faq: 
https://github.com/r-windows/docs/blob/master/faq.md#how-to-install-rjava-on-windows

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rtools and R 4.0.0?

[Jeroen Ooms]

Hevré told me that people from Bioconductor and possibly others are
keeping an eye on this thread so it would be good to post a note here
too.

As of version 4.0.0, the official R for Windows and binary packages
provided via CRAN are built with gcc-8.3.0 from rtools40, as shown in
the CRAN check table:
https://cran.r-project.org/web/checks/check_flavors.html

Instructions and installers for rtools40 are available at the usual
location: https://cran.r-project.org/bin/windows/Rtools/ . Suggestions
for improving the documentation can be posted here:
https://github.com/r-windows/docs








On Tue, Apr 7, 2020 at 6:07 PM Kevin Ushey  wrote:
>
> That's great to see, although I suspect it's still a speculative
> change and could be backed out if any non-trivial issues were
> encountered.
>
> Regardless, I would like to thank R core, CRAN, and Jeroen for all of
> the time that has gone into creating and validating this new
> toolchain. This is arduous work at an especially arduous time, so I'd
> like to voice my appreciation for all the time and energy they have
> spent on making this possible.
>
> Best,
> Kevin
>
> On Tue, Apr 7, 2020 at 7:47 AM Dirk Eddelbuettel  wrote:
> >
> >
> > There appears to have been some progress on this matter:
> >
> > -Note that @command{g++} 4.9.x (as used for @R{} on Windows up to 3.6.x)
> > +Note that @command{g++} 4.9.x (as used on Windows prior to @R{} 4.0.0)
> >
> > See SVN commit r78169 titled 'anticipate change in Windows toolchain', or 
> > the
> > mirrored git commit at
> > https://github.com/wch/r-source/commit/bd674e2b76b2384169424e3d899fbfb5ac174978
> >
> > Dirk
> >
> > --
> > http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] missing binaries in R-devel windows snapshot 78175

[Jeroen Ooms]

On Fri, Apr 10, 2020 at 9:49 AM Martin Maechler
 wrote:
>
> >>>>> Jeroen Ooms
> >>>>> on Fri, 10 Apr 2020 08:54:39 +0200 writes:
>
> > On Fri, Apr 10, 2020 at 2:42 AM Bravington, Mark (Data61,
> > Hobart)  wrote:
> >>
> >> > On Thu, Apr 9, 2020 at 12:44 PM Bravington, Mark
> >> (Data61, Hobart) > 
> >> wrote:
> >> > >
> >> > > The "r-devel snapshot build" 78175 on Windows--- a
> >> dot-exe installer--- seems to be missing a couple of
> >> files in its bin/i386 folder: Rterm.exe and
> >> Rgui.exe. Both are present in its bin/x64 folder (and in
> >> the i386 folder for current R).
> >>
> >> From: Jeroen Ooms  Sent: Thursday, 9
> >> April 2020 21:32 To: Bravington, Mark (Data61, Hobart)
> >> Cc: R-Devel-2 Subject: Re: [Rd] missing binaries in
> >> R-devel windows snapshot 78175
> >>
> >>
> >> > I just tested this and as far as I can tell there are
> >> no missing > files. Are you sure there isn't a local
> >> problem with your system > permissions or antivirus that
> >> is removing the files?
> >>
> >> You're quite right--- my apologies. I've been Cylanced
> >> (without it having the politeness to actually tell me,
> >> g). I tried again with the 78172 build that you used,
> >> and the two dot-exe files are there immediately after
> >> installation--- but 10 minutes later they're gone. One
> >> question below about the md5 stuff, though:
> >>
> >> > Try to verify the md5 of the installer; some enterprise
> >> firewalls are > tampering with downloads:
> >>
> >> >
> >> 
> openssl::md5(url('https://cran.r-project.org/bin/windows/base/R-devel-win.exe'))
> >> >
> >> 
> readLines('https://cran.r-project.org/bin/windows/base/md5sum.txt.R-devel')
> >>
> >> Thanks for the tip. I did that, and both give the same
> >> output. But:
> >>
> >> - If I run 'md5sum' on my local copy of the 78182
> >> installer (ie the file I've just downloaded) I again get
> >> the same signature as per your two lines. (It's not the
> >> firewall, it's Cylance.)
> >>
> >> - Yet if I run 'openssl::md5( )'
> >> I get a different signature!
>
> > I think you're calculating the md5 of the filename, not
> > the file. You need to use:
>
> >   openssl::md5(file("R-devel-win.exe"))
>
> or use base R'stools::md5sum()
>
> md5sum() has existed forever, but needs you to do the
> download.file() step  which is indirectly happening in
> openssl::md5() also.

Not entirely the same way though! The openssl implementation of md5,
sha256, etc calculate the hash directly from the connection stream,
without ever storing the entire file on disk or in memory. It
efficiently hashes arbitrary large data from any readable R connection
(url, file, pipe, etc) using a fixed amount of memory. This is
important for researchers that calculate hashes for very large data
files, where copying the entire data to disk or memory is impossible.

However, in this case it's just a small file, so tools::md5sum() is
simpler indeed :)

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] missing binaries in R-devel windows snapshot 78175

[Jeroen Ooms]

On Fri, Apr 10, 2020 at 2:42 AM Bravington, Mark (Data61, Hobart)
 wrote:
>
> > On Thu, Apr 9, 2020 at 12:44 PM Bravington, Mark (Data61, Hobart)
> >  wrote:
> > >
> > > The "r-devel snapshot build" 78175 on Windows--- a dot-exe installer--- 
> > > seems to be missing a couple of files in its bin/i386 folder: Rterm.exe 
> > > and Rgui.exe. Both are present in its bin/x64 folder (and in the i386 
> > > folder for current R).
>
> From: Jeroen Ooms 
> Sent: Thursday, 9 April 2020 21:32
> To: Bravington, Mark (Data61, Hobart)
> Cc: R-Devel-2
> Subject: Re: [Rd] missing binaries in R-devel windows snapshot 78175
>
>
> > I just tested this and as far as I can tell there are no missing
> > files. Are you sure there isn't a local problem with your system
> > permissions or antivirus that is removing the files?
>
> You're quite right--- my apologies. I've been Cylanced (without it having the 
> politeness to actually tell me, g). I tried again with the 78172 build 
> that you used, and the two dot-exe files are there immediately after 
> installation--- but 10 minutes later they're gone. One question below about 
> the md5 stuff, though:
>
> > Try to verify the md5 of the installer; some enterprise firewalls are
> > tampering with downloads:
>
> >   
> > openssl::md5(url('https://cran.r-project.org/bin/windows/base/R-devel-win.exe'))
> >   
> > readLines('https://cran.r-project.org/bin/windows/base/md5sum.txt.R-devel')
>
> Thanks for the tip. I did that, and both give the same output. But:
>
>  - If I run 'md5sum' on my local copy of the 78182 installer (ie the file 
> I've just downloaded) I again get the same signature as per your two lines. 
> (It's not the firewall, it's Cylance.)
>
>  - Yet if I run 'openssl::md5( )' I get a different 
> signature!

I think you're calculating the md5 of the filename, not the file. You
need to use:

  openssl::md5(file("R-devel-win.exe"))

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] missing binaries in R-devel windows snapshot 78175

[Jeroen Ooms]

On Thu, Apr 9, 2020 at 12:44 PM Bravington, Mark (Data61, Hobart)
 wrote:
>
> The "r-devel snapshot build" 78175 on Windows--- a dot-exe installer--- seems 
> to be missing a couple of files in its bin/i386 folder: Rterm.exe and 
> Rgui.exe. Both are present in its bin/x64 folder (and in the i386 folder for 
> current R).

I just tested this and as far as I can tell there are no missing
files. Are you sure there isn't a local problem with your system
permissions or antivirus that is removing the files?

Try to verify the md5 of the installer; some enterprise firewalls are
tampering with downloads:

  
openssl::md5(url('https://cran.r-project.org/bin/windows/base/R-devel-win.exe'))
  readLines('https://cran.r-project.org/bin/windows/base/md5sum.txt.R-devel')

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rtools and R 4.0.0?

[Jeroen Ooms]

On Mon, Apr 6, 2020 at 9:39 AM Gabriel Becker  wrote:
>
> Hi Kevin,
>
> On Wed, Apr 1, 2020 at 9:36 PM Kevin Ushey  wrote:
>
> > Hello,
> >
> > FWIW, I'm excited at the prospect at seeing a new toolchain for
> > Windows, since it would imply support for C++17 and so it would become
> > easier for CRAN packages to depend on the newer C++ standard.
> >
>
> One thing to keep in mind (having been the R installation owner in such a
> place for multiple years) is that many coproprate or otherwise controlled
> compute environments may not have access to a c++17 compiler on their
> servers so making it easy for packages to rely on that is not purely
> beneficial to all parts of the R community.

No, you're missing an important point here. On Windows, the toolchain
version is tied to the version of R and we try to keep supporting at
least one or two previous versions of R. So this means we always need
to support the legacy toolchain for a while as well.

Hence if we switch Windows to gcc-8 for R 4.0, we still rely on
gcc-4.9 for continued support of R 3.3-3.6. This lag is what is making
the maintenance of windows system libraries painful, and why we need
to plan ahead. This is different from Linux where version of the
compiler is given by the OS and not tied to the version of R.

If we would miss the boat again, and R 4.0 on Windows would stick with
gcc-49, this means we need to keep supporting gcc-49 as long as we
want to support R-4.0, which is at least 2022 or 2023. This would be
pretty bad. Even currently the latest versions of important system
libraries used by R packages (e.g. the gdal stack) require recent
compilers and cannot be built anymore with gcc-49. As more C++
projects are adopting C++14/17, we can no longer update these system
libraries, missing out on all upstream fixes and advances. This would
seriously decrement the quality of the R ecosystem.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rtools and R 4.0.0?

[Jeroen Ooms]

On Mon, Apr 6, 2020 at 9:15 AM Prof Brian Ripley  wrote:
>
> On 02/04/2020 05:35, Kevin Ushey wrote:
> > Hello,
> >
> > Has a decision been made yet as to whether R 4.0.0 on Windows is going
> > to be built using the new gcc8 toolchain (described at
> > https://cran.r-project.org/bin/windows/testing/rtools40.html)?
>
> Short answer: 'no'.
>
> >>From the sidelines, I can see that the toolchain is being used to
> > build and test packages on CRAN; if there are any remaining issues
> > that I can help to try and run down (either in R or any CRAN packages)
> > I'd be happy to try and help.
>
> It is still under consideration, but resource constraints are biting
> hard as people's lives get more complicated.

Given the amount of work that has gone into this and the importance of
an up-to-date toolchain for maintaining well-functioning and current
system libraries and R packages, I think it should be considered to
delay the R 4.0 release over this until everyone is ready.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] CRAN Package Check Results - No protocol specified (OS X only)

[Jeroen Ooms]

On Tue, Feb 18, 2020 at 1:29 AM Dominic Comtois
 wrote:
>
> Hello,
>
> On my package's check results (
> https://cran.r-project.org/web/checks/check_results_summarytools.html), I
> see a bunch of warnings with "No protocol specified" messages. This happens
> only with OS X, and I can't reproduce them when actually building on a Mac,
> nor with rhub::check_on_macos(). I can't really make sense out of them, and
> they are all over the place. Can they simply be ignored?

This happens when your package calls capabilities(), which as a side
effect initiates xquartz. It seems the MacOS server has certain
permission restrictions that cause X11 to print this warning.

I think you can safely ignore it.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] Development version of R fails tests and is not installed

[Jeroen Ooms]

On Sat, Feb 8, 2020 at 9:27 AM Berwin A Turlach
 wrote:
>
> G'day all,
>
> I have daily scripts running to install the patched version of the
> current R version and the development version of R on my linux box
> (Ubuntu 18.04.4 LTS).
>
> The last development version that was successfully compiled and
> installed was "R Under development (unstable) (2020-01-25 r77715)" on
> 27 January.  Since then the script always fails as a regression test
> seems to fail.  Specifically, in the tests/ subdirectory of my build
> directory I have a file reg-tests-1d.Rout.fail which ends with:
>
> > ## more than half of the above were rounded *down* in R <= 3.6.x
> > ## Some "wrong" test cases from CRAN packages (partly relying on wrong R <= 
> > 3.6.x behavior)
> > stopifnot(exprs = {
> + all.equal(round(10.7775, digits=3), 10.778, tolerance = 1e-12) # even 
> tol=0, was 10.777
> + all.equal(round(12345 / 1000,   2), 12.35 , tolerance = 1e-12) # even 
> tol=0, was 12.34 in Rd
> + all.equal(round(9.18665, 4),9.1866, tolerance = 1e-12) # even 
> tol=0, was  9.1867
> + })
> Error: round(10.7775, digits = 3) and 10.778 are not equal:
>   Mean relative difference: 9.27902e-05
> Execution halted
>
> This happens while the 32bit architecture is installed,  which is a bit
> surprising as I get the following results for the last installed
> version of R's development version

There are two independent, but slightly related issues here:

First, as Martin already explained, the round() function was recently
improved, and some very strict tests were added to confirm the new
behavior. That explains why you see different round() results in R 4.0
from R 3.6.2. The bugzilla thread explains why:
https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17668

The second issue has to do with numeric precision on 32-bit systems,
which is why I think you are getting this error. We ran into the same
problem on Windows where results on 32-bit are slightly off, including
(but not limited to) edge-cases in rounding. This has always been the
case, but the 32-bit inaccuracies have increased for recent versions
of GCC.

In general, the main difference in float precision between i686 and
x86_64 could come from whether it uses x87 (with 80 bit floats as
intermediates, as long as all intermediates are stored in registers)
or sse2 for general math. Depending on what the tests do, you can get
test failures (i.e. different results) if intermediates use different
precision, if the test reference is calculated assuming rounding all
intermediates to a certain length between each step.

The solution: to get the same results on 32-bit as on 64-bit, you need
to build R with these extra gcc flags: -mfpmath=sse -msse2. As
explained in 
https://gcc.gnu.org/onlinedocs/gcc-8.3.0/gcc/x86-Options.html#x86-Options
the -mfpmath=sse is the default for x86-64 but not for i686. As of
r77719 we have made sse the default on Windows and now we get
consistent results on 32-bit and 64-bit, including the round() edge
cases.

I think the intention was to add something similar in R's autoconf
script to enable sse on 32-bit unix systems, but seemingly this hasn't
happened. For now I think you should be able to make your 32-bit
checks succeed if you build R with CFLAGS=-mfpmath=sse -msse2.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Can't build R 3.6.1 or 3.5.3 on Ubuntu 18.04 LTS

[Jeroen Ooms]

On Fri, Nov 22, 2019 at 2:58 PM Saren Tasciyan  wrote:
>
> Hi,
>
> I am trying to build a more recent version (3.6.1 or 3.5.3) of R on
> Ubuntu 18.04 LTS.

Are you aware you can get up-to-date binaries either from PPA or CRAN:

  sudo add-apt-repository ppa:marutter/rrutter
  sudo apt-get install r-base-dev

> and I ran ./configure
> During "make", I get the following error:
> platform.o: In function `do_eSoftVersion':
> /home/user/Desktop/R-3.6.1/src/main/platform.c:3129: undefined reference
> to `u_getVersion_58'

Such linking errors indicate you are not linking against the correct
library that you compiled against. In this case, there seems to be a
version conflict because you compiled against version 58 of ICU, but
Ubuntu Bionic actually has ICU60:
https://packages.ubuntu.com/bionic/libicu-dev

So the question is how do you end up with ICU58 headers? I am guessing
your either installed a custom libicu in /usr/local/, or you upgraded
from an older version of ubuntu but haven't upgraded libicu-dev.
Either way the solution is to clean up your system and make sure there
is only one version of libicu.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rtools contains Python interpreter(s), and six copies?

[Jeroen Ooms]

On Fri, Aug 2, 2019 at 12:12 AM Abby Spurdle  wrote:
>
> I've just discovered that Rtools (on Windows) contains Python
> interpreter(s).

A minimal build of python was included as a dependency of gdb in
Rtools 3.3 and up.

> I'm assuming that Python is required to build R packages, on all operating
> systems.

Please don't assume but read the documentation (preferably before posting).

> Also, by my count, Rtools contains six Python interpreters.
> I've miscounted, I hope...

We have a dual toolchain so there is a 32 and a 64 bit build of
python2.7.exe. The other names (python.exe and python2.exe) are
symlinks that python automatically creates. These are not in any
location that is ever on the path so nothing to worry about.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Offer zip builds

[Jeroen Ooms]

On Tue, Jun 4, 2019 at 5:40 PM Steven Penny  wrote:
>
> Theres nothing nefarious here. It would allow people to use the R environment
> without running an installer. If someone is a new user they may want to try
> R out, and installers can be invasive as they commonly:
>
> - copy files to install dir
> - copy files to profile dir
> - set registry entries
> - set environment variables
> - set start menu entries
>
> and historically uninstallers have a bad record of reverting these changes.
> should not put this burden upon new users or even having them resort to 
> virtual
> machine to avoid items above. having a ZIP file allows new users to run the
> R environment, then if they like it perhaps they can run the installer going
> forward.

This is a valid suggestion, but probably impossible to do reliably.
Most installers (the R one is completely open source btw) perform
those steps for a reason. It is great if software can be installed
simply by extracting a zip file somewhere, but if this is what you
desire, you're using the wrong operating system.

We only offer official installation options that work 100% reliably
and I don't think this can be accomplished with a zip file. For
example a zip file won't be able to set the installation location in
the registry, and hence other software such as RStudio won't be able
to find the R installation. Also a zip installation might mix up
package libraries from different R versions (which is bad), or users
might expect they can upgrade R by overwriting their installation with
a new zip (also bad). Hence I'm afraid offering such alternative
installation options would open a new can of worms with bug reports
from Windows users with broken installations, or packages that don't
work as expected.

As for alternatives, 'rportable' and 'innoextract' have already been
mentioned if you really just want to dump the files from the
installer, if that works for you. Another popular option to install
(any) Windows software without manually running installers is using
chocolatey, for example:

  choco install miktex
  choco install r.project

This will still indirectly use official installers, but the installers
have been verified as "safe" by external folks and the installation is
completely automated. Perhaps that's another compromise you could live
with.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] writing Unicode text to the Windows clipboard

[Jeroen Ooms]

On Fri, May 24, 2019 at 12:06 AM Jennifer Bryan  wrote:
>
> Hello,
>
> I'm interested in moving text from and to the clipboard that cannot
> necessarily be represented in the native encoding. So, really, this is
> about Windows.
>
> I can successfully read from the clipboard by specifying the format that
> corresponds to unicode text.
>
> From R >=2.7.0, it seems you should also be able to write unicode text
> to the Windows clipboard.
>
> https://github.com/wch/r-source/blob/5a156a0865362bb8381dcd69ac335f5174a4f60c/src/gnuwin32/CHANGES0#L535-L536

Thanks!

I tested this and can confirm that this patch works. With this change,
any (non ascii) unicode text now properly copies and pastes from/to
the Windows clipboard.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Inno Setup 6.0.2 fails before creating exe file on Windows (R-3.6.0)

[Jeroen Ooms]

Multiple people have now reported that the R installer does not build
with InnoSetup 6 (released last week). The inno log shows this error:

  Error on line 12 in src\gnuwin32\installer\R.iss: Minimum NT version
specified by MinVersion must be at least 6.0. (Windows 2000/XP/Server
2003 are no longer supported.)
  Compile aborted.

Hence to fix this, the "MinVersion" parameter in
src/gnuwin32/installer/header1.iss should be increased from 0,5.0 to
0,6.0. I confirmed this resolves the problem. Could somebody commit
this please?

See also: http://www.jrsoftware.org/is6help/index.php?topic=setup_minversion



On Sun, Apr 28, 2019 at 11:07 AM Avraham Adler  wrote:
>
> I am working on compiling R-3.6.0 for Windows 10 64bit using rtools40
> (beta 11). I had also installed the most recent update of Inno setup,
> which is now 6.0.2.With that version, `make risntaller` fails at the
> call to ""C:/R/Inno/iscc" R.iss > R-3.6.0.log 2>&1" and just exits,
> pointing to line 175 of the makefile which is:
>
> $(RPREFIX)-win.exe: R.iss
> "$(ISDIR)/iscc" R.iss > $(RPREFIX).log 2>&1
>
> Reinstalling Inno Setup 5.6.1 does allow the exe file to be created.
>
> Thank you,
>
> Avi
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R 3.5.3 and 3.6.0 alpha Windows bug: UTF-8 characters in code are simplified to wrong ones

[Jeroen Ooms]

On Wed, Apr 10, 2019 at 5:45 PM Duncan Murdoch  wrote:
>
> On 10/04/2019 10:29 a.m., Yihui Xie wrote:
> > Since it is "technically easy" to disable the best fit conversion and
> > the best fit is rarely good, how about providing an option for
> > code/package authors to disable it? I'm asking because this is one of
> > the most painful issues in packages that may need to source() code
> > containing UTF-8 characters that are not representable in the Windows
> > native encoding. Examples include knitr/rmarkdown and shiny. Basically
> > users won't be able to knit documents or run Shiny apps correctly when
> > the code contains characters that cannot be represented in the native
> > encoding.
>
> Wouldn't things be worse with it disabled than currently?  I'd expect
> the line containing the "ř" to end up as NA instead of converting to "r".

I don't think it would be worse, because in this case R would not
implicitly convert strings to (best fit) latin1 on Windows, but
instead keep the (correct) string in its UTF-8 encoding. The NA only
appears if the user explicitly forces a conversion to latin1, which is
not the problem here I think.

The original problem that I can reproduce in RGui is that if you enter
 "ř" in RGui, R opportunistically converts this to latin1, because it
can. However if you enter text which can definitely not be represented
in latin1, R encodes the string correctly in UTF-8 form.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R 3.5.3 and 3.6.0 alpha Windows bug: UTF-8 characters in code are simplified to wrong ones

[Jeroen Ooms]

On Wed, Apr 10, 2019 at 12:19 PM Tomáš Bořil  wrote:
>
> Minimalistic example:
> Let's type "ř" (LATIN SMALL LETTER R WITH CARON) in RGui console:
> > "ř"
> [1] "r"
>
> Although the script is in UTF-8, the characters are replaced by
> "simplified" substitutes uncontrollably (depending on OS locale). The
> same goes with simply entering the code statements in R Console.
>
> The problem does not occur on OS with UTF-8 locale (Mac OS, Linux...)

I think this is a "feature" of win_iconv that is bundled with base R
on Windows (./src/extra/win_iconv). The character from your example is
not part of the latin1 (iso-8859-1) set, however, win-iconv seems to
do so anyway:

> x <- "\U00159"
> print(x)
[1] "ř"
> iconv(x, 'UTF-8', 'iso-8859-1')
[1] "r"

On MacOS, iconv tells us this character cannot be represented as latin1:

> x <- "\U00159"
> print(x)
[1] "ř"
> iconv(x, 'UTF-8', 'iso-8859-1')
[1] NA

I'm actually not sure why base-R needs win_iconv (but I'm not an
encoding expert at all). Perhaps we could try to unbundle it and use
the standard libiconv provided by the Rtools toolchain bundle to get
more consistent results.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Is libtiff >= 4.0.0 now required by R for TIFF support?

[Jeroen Ooms]

On Mon, Feb 25, 2019 at 11:11 AM Bjørn-Helge Mevik
 wrote:
>
> We recently discovered that since R 3.3.0, on our CentOS 6 based
> cluster, R gets built without TIFF support.  The last version where TIFF
> support was built, was 3.2.5.

It looks like the official R from EPEL6 also lacks tiff support:

  yum install epel-release
  yum install R-devel
  R -e 'capabilities()'

> capabilities()
   jpeg pngtiff   tcltk X11aqua
   TRUETRUE   FALSETRUE   FALSE   FALSE
   http/ftp sockets  libxmlfifo  cledit   iconv
   TRUETRUETRUETRUE   FALSETRUE
NLS profmem   cairo ICU long.double libcurl
   TRUE   FALSETRUETRUETRUETRUE

So your hypothesis that autoconf doesn't find libtiff < 4 is probably correct.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Trying to compile R 3.5.2 - 32 bit R - on Windows 10 64 bit - with ICU support

[Jeroen Ooms]

On Sat, Feb 16, 2019 at 11:13 AM Andre Mikulec
 wrote:
>
> Hi,
>
> I am trying to compile R with ICU support.

Are you aware that the default R for Windows installation from CRAN
has ICU support?

> I am following
> https://cran.r-project.org/doc/manuals/R-admin.html#Building-from-source

These instructions are unfortunately outdated. The scripts and
instructions used to build the current version of R for Windows can be
found here: https://github.com/rwinlib/base#readme

> I have downloaded and extracted
> https://www.stats.ox.ac.uk/pub/Rtools/goodies/ICU_531.zip

That build of ICU was for the previous toolchain (up to R 3.2.5). You
need to use the ICU build which is included with Rtools (if you
checked the icu checkbox during the Rtools installation). The scripts
mentioned above will do this automatically.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Issue when building R-parched_2018-12-26

[Jeroen Ooms]

On Wed, Dec 26, 2018 at 8:24 PM Avraham Adler 
wrote:

> Hello.
>
> Building R-patched from source on Win64 using current Rtools 3.5.0.4, I’m
> getting the following notes indicating an extra left brace somewhere; or is
> the problem on my end?
>

This is a warning (not an error) in the texinfo script that has started to
appear with the latest version of perl. It is safe to ignore the warning,
but it has been fixed in rtools40:
https://github.com/r-windows/rtools-base/commit/d1631cc9646121f580742a29b2ee3c3b8a569685

I haven't back-ported this to rtools35 because I'm still hopeful that we
will upgrade to rtools40, soonish...

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] OfficeScan deletes Rterm as malware

[Jeroen Ooms]

On Thu, Dec 20, 2018 at 2:52 PM Roland Fuß  wrote:
>
> I'm not sure if the problem is actually with R but thought I should
> report this anyway.
>
> After Peter's email regarding the R 3.5.2 release today, I installed the
> Windows version right away (directly from CRAN and not from a mirror).
>
> Unfortunately, my institute's AV sofware TrendMicro OfficeScan 12.0.5147
> Service Pack 1 stops and deletes Rterm.exe when running Rcmd.exe INSTALL
> with a local source package. It reports "OfficeScan detected a Behavior
> Monitoring policy violation and blocked the offending process(es)." and
> "Unauthorized File Encryption" by Rterm.exe.

I can't think of anything that has changed between R 3.5.1 and 3.5.2.
What does "Behavior Monitoring policy violation" mean? Does it say
what sort of policy?

A quick search reveals that the TrendMicro "Unauthorized File
Encryption" message is a frequent false positive for many different
software programs, including this thread at the rstudio forum:
https://support.rstudio.com/hc/en-us/community/posts/208171047-rsession-exe-flagged-as-virus

Perhaps it's just because you're one of the first people to install
this version, and your AV might use some "smart" learning system such
that the false positive will automatically disappear after a few more
people have installed and whitelisted the new R binaries.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Change windows installer default to only install 64 bit R

[Jeroen Ooms]

On Fri, Nov 9, 2018 at 5:47 PM Jim Hester  wrote:
> We can remove much of this ambiguity by changing the default choice in
> the installer to only install the 64 bit version in the installer. If
> users do need the 32 bit version it is still simple for them to
> install it by checking the appropriate box during installation.

I tested the patch and confirm that this works as intended. By
default, the R installer will leave the "32-bit Files" unchecked on
Windows 64 machines.

I also think this proposal makes sense. Most users do not need 2
versions of R on their machine, and it is a common source of problems.
I agree with Jim that installing the 32bit version of R on 64bit
Windows would better be opt-in rather than default.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] segfault issue with parallel::mclapply and download.file() on Mac OS X

[Jeroen Ooms]

On Thu, Oct 4, 2018 at 6:12 PM Tomas Kalibera  wrote:
>
>
> Thanks for the report, but unfortunately I cannot reproduce on my system
> (either macOS nor Linux, from the command line) to debug. Did you run
> this in the command line version of R?

It depends on which version of MacOS that you are using and
specifically which TLS back-end curl has been configured with. When
libcurl uses DarwinSSL, it may crash when opening HTTPS connections in
a fork because CoreFoundation is not fork-safe. OTOH when using
OpenSSL or LibreSSL for TLS, you usually get away with forking (though
it's still bad practice).

The standard version of libcurl that ships with MacOS was using
CoreFoundation until 10.12 but starting 10.13 they switched to
LibreSSL in order to support HTTP/2. See curl --version or
curl::curl_version() for your local config. Don't count in this
though, Apple might switch back to the fork-unsafe DarwinSSL once they
support ALPN, which is needed for HTTP/2.

As Gabor already suggested, libcurl has built-in systems for
concurrent connections. The curl package exposes this via multi_add
function. Not only is this safer than forking, it will be much faster
because it takes advantage of HTTP keep-alive and when supported it
uses HTTP2 multiplexing which allows efficiently performing thousands
of concurrent HTTPS requests over a single TCP connection.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] substitute() on arguments in ellipsis ("dot dot dot")?

[Jeroen Ooms]

On Sun, Aug 12, 2018 at 10:00 PM, Henrik Bengtsson
 wrote:
> Hi. For any number of *known* arguments, we can do:
>
> one <- function(a) list(a = substitute(a))
> two <- function(a, b) list(a = substitute(a), b = substitute(b))
>
> and so on. But how do I achieve the same when I have:
>
> dots <- function(...) list(???)
>
> I want to implement this such that I can do:
>
>> exprs <- dots(1+2)
>> str(exprs)
> List of 1
>  $ : language 1 + 2
>
> as well as:
>
>> exprs <- dots(1+2, "a", rnorm(3))
>> str(exprs)
> List of 3
>  $ : language 1 + 2
>  $ : chr "a"
>  $ : language rnorm(3)
>
> Is this possible to achieve using plain R code?

You could use match.call, for example:

  dots <- function(...) match.call(expand.dots = FALSE)[['...']]

Note that this returns a pairlist, so if you want an ordinary list you
should wrap it in as.list()

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] undefined symbol: cholmod_factorize_p

[Jeroen Ooms]

On Fri, Jul 6, 2018 at 11:57 PM, Göran Broström  wrote:
> I am installing R_3.5.1 from source on ubuntu 18.04, and 'config' + 'make'
> gives me (at the end)

Probably something went wrong earlier in the build when compiling the
Matrix package. However it's much easier to install R form the
binaries:

  sudo add-apt-repository -y ppa:marutter/rrutter3.5
  sudo apt-get update
  sudo apt-get install r-base

If you really want to build from source, have a look at which flags
and dependencies are used in the r-base source package on
Debian/Ubuntu.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] potential Bug in R 3.5.0

[Jeroen Ooms]

On Sun, Jun 24, 2018 at 8:41 PM, Jochen Wirsing  wrote:
> Hello,
> I am using R under Win7 and Antergos. After installing R 3.5.0 I ran into
> several issues regarding gfortran not working properly and packages not
> installing properly when requiring dependencies (which is quite often the case
> when you have to reinstall all packages for the new R version).
> Unfortunately, I cannot reproduce the gfortran issue at the moment (occuring
> under Antergos), but regarding the depency-issue, https://pastebin.com/
> 0nU5n3pH shows one instance of the problem, when using "tidyverse". Adding
> "dependencies = TRUE" does not help.

It looks like R thinks those packages (‘dbplyr’, ‘rvest’, ‘xml2’) are
already installed, but is unable to load them. I have never used
'Antergos' but could it be that those packages were installed with R
3.4 and therefore stopped working after you upgraded to R 3.5? You
need to rebuild all R packages after you update to R. The easiest way
is to run:

  update.packages(ask = F, checkBuilt = T)

Also check your .libPaths() to make sure that there are no old
packages left behind in older directories.

If this is not the issue, try finding out where those packages live
using e.g. find.packages("xml2") and see what error you get when
trying to load them with library("xml2").

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] download.file does not process gz files correctly (truncates them?)

[Jeroen Ooms]

On Thu, May 3, 2018 at 2:42 PM, Henrik Bengtsson
 wrote:
> Use mode="wb" when you download the file. See
> https://github.com/HenrikBengtsson/Wishlist-for-R/issues/30.
>
> R core, and others, is there a good argument for why we are not making this
> the default download mode? It seems like a such a simple fix to such a
> common "mistake".

I'd like to second this feature request. This default behaviour is
unexpected and often leads to r scripts that were written on
mac/linux, to produce corrupted files on windows, checksum mismatches,
etc.

Even for text files, the default should be to download the file as-is.
Trying to "fix" line-endings should be opt-in, never the default.
Downloading a file via a browser or ftp client on windows also doesn't
change the file, why should R?


On Thu, May 3, 2018 at 3:02 PM, Duncan Murdoch  wrote:
> Many downloads are text files (HTML, CSV, etc.), and if those are downloaded
> in binary, a Windows user might end up with a file that Notepad can't
> handle, because it would have Unix-style line endings.

True but I don't think this is relevant. The same holds e.g. for the R
files in source packages, which also have unix line endings. Most
Windows users will use an actual editor that understands both types of
line endings, or can convert between the two.

Downloading-file should do just that.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] data.table not available as win binary for R 3.5

[Jeroen Ooms]

On Tue, Apr 24, 2018 at 7:26 AM, Joris Meys  wrote:
>
> Dear all,
>
> to my astonishment data.table cannot be installed on R 3.5 Windows. When
> checking the package page, the Windows binary is available for download.


The package check page for data.table shows that is currently failing
CMD check. As a precaution, CRAN does not publish binaries for
packages that do not pass check, so I think this is why it seems
unavailable.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Problem with R_registerRoutines

[Jeroen Ooms]

On Windows this warning may be a false positive if R cannot find
"objdump.exe" which is required for this check. I think this is
actually a bug in R because it should be looking for "objdump.exe"
inside BINPREF (where gcc is) rather than on the PATH.

Can you check if you get the same warning if you upload the package to
https://win-builder.r-project.org ?






On Fri, Feb 23, 2018 at 10:28 AM,   wrote:
> Dear list,
>
> I am trying to update a package to pass the CRAN-checks.
> But I am struggling with the following note:
>
> File 'psgp/libs/i386/psgp.dll':
>   Found no calls to: 'R_registerRoutines', 'R_useDynamicSymbols'
> File 'psgp/libs/x64/psgp.dll':
>   Found no calls to: 'R_registerRoutines', 'R_useDynamicSymbols'
>
> It is good practice to register native routines and to disable symbol
> search.
>
>
> I did already run:
> tools::package_native_routine_registration_skeleton(".")
> This gave me some code, including a function R_init_psgp, which includes 
> calls to the functions above, and also the names of the C++ functions to be 
> called from R.
> I first saved this code in registerDynamicSymbol.c and added .registration = 
> TRUE to useDynLib in the NAMESPACE file.
> I still get the error above. As I saw that the file has different names in 
> other packages, I have also tried to save it psgp_init.c, and in init.cpp, 
> still with the same error message.
>
> I have read the relevant part of the R extensions manual, but could not find 
> anything that could help me with this problem.
> I have had a look at the similar files in other packages (including one of my 
> own, which works), and the initialization seems fine to me.
> There is surely something I have overlooked, is anyone able to give me a hint 
> to where I might look? The code is in C++, not sure if that could have 
> anything to do with the problem?
>
> Thanks,
> Jon
>
>
>
>
>
>
>
>
>
>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Setting the path to Rtools for package compilation on Windows

[Jeroen Ooms]

On Tue, Feb 13, 2018 at 7:58 PM, Peter Langfelder
<peter.langfel...@gmail.com> wrote:
> On Tue, Feb 13, 2018 at 2:20 AM, Jeroen Ooms <jeroeno...@gmail.com> wrote:
>
>> Thanks for your question. Your logs show that 'gcc' is not found. As
>> of R 3.3 you need to set the path to the compiler using the BINPREF
>> variable. This is because we ship two separate versions of gcc, one
>> targeting win32 and one targeting win64. I am not sure what your
>> rtools installation looks like, but could you try setting this
>> environment variable:
>>
>>   BINPREF="M:/R/Rtools-3.4/mingw_$(WIN)/bin/"
>>
>> I think this will do the job.
>
> Thanks, that indeed did the trick. May I suggest that this hint be
> also included in "REMAINING TASKS" section of the file Rtools.txt that
> is part of the Rtools distribution?  The R installation manual does
> mention BINPREF and BINPREF64 but I missed that part...

Glad it worked. Actually BINPREF64 only exists when building R itself.
For the R user there is no BINPREF64. There is only BINPREF which has
to point to a 32bit gcc when R runs in 32bit, and a 64bit compiler
when R runs in 64bit. But if you compile R packages you need both at
the same time. Hence the "$(WIN)" variable in the BINPREF above.

Yes the rtools setup is a bit convoluted, and we're currently
discussing ways to improve this. Thanks for your suggestion.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R Compilation gets stuck on Windows 64

[Jeroen Ooms]

On Tue, Feb 13, 2018 at 2:18 PM, Indrajit Sen Gupta
 wrote:
> In the file MkRules.local.in, I see the line: USE_ATLAS = NO which I believe
> needs to be changed to YES. But how do I specify the BLAS file
> libopenblas_haswell-r0.2.20.a and its location?

I have never done this, but a good starting point is searching the R
source code for how this works:
https://github.com/wch/r-source/search?utf8=✓=USE_ATLAS

>From the extra/blas/Makefile.win file it looks like you may need to
rename libopenblas_haswell-r0.2.20.a to libatlas.a and point to this
directory via the variable ATLAS_PATH in your MkRules.local.in?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R Compilation gets stuck on Windows 64

[Jeroen Ooms]

On Tue, Feb 13, 2018 at 12:22 PM, Indrajit Sen Gupta
 wrote:
> Hi Avraham,
>
> I tried with the patched version. The same error message.
>
> gcc -std=gnu99 -m64 -shared -s -mwindows -o R.dll R.def console.o dynload.o
> editor.o embeddedR.o extra.o malloc.o opt.o pager.o preferences.o psignal.o
> rhome.o rt_complete.o rui.o run.o shext.o sys-win32.o system.o dos_wglob.o
> dllversion.o ../main/libmain.a ../appl/libappl.a ../nmath/libnmath.a
> getline/gl.a ../extra/xdr/libxdr.a ../extra/intl/libintl.a
> ../extra/trio/libtrio.a ../extra/tzone/libtz.a ../extra/tre/libtre.a
> -fopenmp -L. -lgfortran -lquadmath -lRblas -L../../lib -lRgraphapp -lRiconv
> -lcomctl32 -lversion -L"D:/R64/extsoft"/lib/x64 -lpcre -lz -lbz2 -llzma
> -L"D:/home/ICU"/lib/x64 -lsicuin -lsicuuc -lsicudt -lstdc++
> D:/Rtools/mingw_64/bin/../lib/gcc/x86_64-w64-mingw32/4.9.3/../../../../x86_64-w64-mingw32/bin/ld.exe:
> cannot find -lRgraphapp
> collect2.exe: error: ld returned 1 exit status
> make[4]: *** [R.dll] Error 1
> make[3]: *** [../../bin/x64/R.dll] Error 2
> make[2]: *** [rbuild] Error 2
> make[1]: *** [all] Error 2
> make: *** [distribution] Error 2
>
>
> Would it be possible for you to share your MkRules.local and Makefile.win
> files?


Hi Indrajit

As somebody above already mentioned, the full build script as well as
MkRules.local that we use for the CRAN releases of R for windows are
available from https://github.com/rwinlib/base

As is explained in the repository readme, if you have the required
dependencies (rtools, miktex innosetup, strawberry perl) all you need
to do is run the build-r-devel.bat script from the root of the
repository.

Once you got this to work, you can adapt it to your needs.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Setting the path to Rtools for package compilation on Windows

[Jeroen Ooms]

On Tue, Feb 13, 2018 at 7:45 AM, Peter Langfelder
 wrote:
> Hi all,
>
> I'm trying to set up the Windows Rtools toolset for building packages
> with compiled code. I installed for Windows R-3.4.3 from CRAN and
> installed Rtools-3.4 in a custom location M:\R\R-3.4.3 and
> M:\R\Rtools-3.4
>
> Following the instructions, in shell, I set
> Path=M:\R\Rtools-3.4\bin;M:\R\Rtools-3.4\gcc-4.6.3\bin;M:\R\R-3.4.3\bin;...
> (the ... are other paths irrelevant for R/Rtools).

Thanks for your question. Your logs show that 'gcc' is not found. As
of R 3.3 you need to set the path to the compiler using the BINPREF
variable. This is because we ship two separate versions of gcc, one
targeting win32 and one targeting win64. I am not sure what your
rtools installation looks like, but could you try setting this
environment variable:

  BINPREF="M:/R/Rtools-3.4/mingw_$(WIN)/bin/"

I think this will do the job.

On Tue, Feb 13, 2018 at 10:21 AM, Tomas Kalibera
 wrote:
> Thanks for the report - this has been already reported as bug 17376, it is
> caused by scripts that build the Windows binaries and by now has been fixed
> in R-patched and R-devel snapshot builds. So as a solution that works now I
> would recommend using R-patched.

This issue is unrelated, I doubt your advice will solve anything. The
only thing that 17376 does is add c:/rtools/bin to the default path.
But this dir does not exist for this user, so it is ignored by windows
when searching the PATH.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] LCONS undefined for R_NO_REMAP

[Jeroen Ooms]

Rinternals.h has:

#define CONS(a, b) cons((a), (b))
#define LCONS(a, b) lcons((a), (b))

However these are undefined when we compile with -DR_NO_REMAP. Maybe
it's safer to define these using Rf_cons() and Rf_lcons() instead?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Rcpp - Linking to DLL from another package?

[Jeroen Ooms]

On Mon, Dec 18, 2017 at 2:40 PM, Dirk Eddelbuettel  wrote:
>
> (Moderately wrong list: r-package-devel for packaging questions, rcpp-devel
> for Rcpp questions)
>
> On 18 December 2017 at 13:24, Stravs, Michael wrote:
> | I am trying to make a package B that extends another package A. Package A 
> uses Rcpp, and I want to extend a class X used there.

Another, perhaps simpler example is the xslt package which is an
extension for the xml2 package. The xml2 package exposes its object
types (via ./inst/include/xml2_types.h) so that the xslt package get
to the underlying libxml2 objects.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] check does not check that package examples remove tempdir()

[Jeroen Ooms]

On Thu, Nov 9, 2017 at 1:43 AM, William Dunlap via R-devel
 wrote:
> I was looking at the CRAN package 'bfork-0.1.2', which exposes the Unix
> fork() and waitpid() calls at the R code level, and noticed that the help
> file example for bfork::fork removes R's temporary directory, the value of
> tempdir().   I think it happens because the forked process shares the value
> of tempdir() with the parent process and removes it when it exits.

This has come up a few times:

 - https://stat.ethz.ch/pipermail/r-devel/2017-February/073748.html
 - https://stat.ethz.ch/pipermail/r-devel/2017-April/074149.html

You may want to read up on the latter discussion.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] tibbles are not data frames

[Jeroen Ooms]

On Tue, Sep 26, 2017 at 11:56 AM, Gábor Csárdi  wrote:
>
> On Tue, Sep 26, 2017 at 10:35 AM, Joris Meys  wrote:
> > I don't like the dropping of dimensions either. That doesn't change the
> > fact that a tibble reacts different from a data.frame. So tibbles do not
> > inherit correctly from the class data.frame, and it can thus be argued that
> > it's against OOP paradigms to pretend tibbles inherit from the class
> > data.frame.
>
> I have yet to see an OOP system in which a subclass cannot override the 
> methods
> of its superclass. Not only is this in line with OOP paradigms, it is
> actually one of the essential OOP features.

Not if this compromises type safety. Formal OOP languages enforce the
signature matches when you override a method. The fact that R is
dynamically typed puts this responsibility at the developer. The fact
that tibble [ returns a data frame where it's parent class returns an
atomic vector violates this principle, resulting in the obvious type
errors where tibbles are used as data frames.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] R-devel object header changes that require reinstalling packages

[Jeroen Ooms]

On Tue, Sep 12, 2017 at 8:28 PM,   wrote:
> (https://svn.r-project.org/R/branches/ALTREP/ALTREP.html outlines the
> framework).

Thank you for the nice writeup, hope this makes it into the R journal.
The spelling package finds a few typos:

> spelling::spell_check_files('ALTREP.md', lang = 'en_US')

  WORD  FOUND IN
abstrct   altrep.md:39
attemps   altrep.md:776
defernaltrep.md:767
hanling   altrep.md:699
representaitons   altrep.md:21

 (omitting false positives)

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R Configuration Variable: Maximum Memory Allocation per R Instance

[Jeroen Ooms]

On Sun, Sep 17, 2017 at 12:39 AM, Juan Telleria  wrote:
> Dear R Developers,
>
> In the same way that MySQL/MariaDB's Engine InnoDB or MyISAM/Aria have the
> innodb_buffer_pool_size or the key_buffer_size for setting the maximum
> amount of RAM which can be used by a Server Instance.

Memory is not controlled by R itself because packages may malloc()
directly. However most operating systems have features to limit
resources of a given process. The CRAN package 'unix' has wrappers for
posix setrlimit [1] e.g. unix::rlimit_as() limits address space. This
works pretty well, however I found that the way memory is managed and
counted varies a lot per OS and malloc implementation.

You can also set rlimits on a single evaluation via the rlimit
parameter in sys::eval_safe().


[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/getrlimit.html

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] readLines() segfaults on large file & question on how to work around

[Jeroen Ooms]

On Sat, Sep 2, 2017 at 8:58 PM, Jennifer Lyon  wrote:
> I have a 2.1GB JSON file. Typically I use readLines() and
> jsonlite:fromJSON() to extract data from a JSON file.

If your data consists of one json object per line, this is called
'ndjson'. There are several packages specialized to read ndjon files:

 - corpus::read_ndjson
 - ndjson::stream_in
 - jsonlite::stream_in

In particular the 'corpus' package handles large files really well
because it has an option to memory-map the file instead of reading all
of its data into memory.

If the data is too large to read, you can preprocess it using
https://stedolan.github.io/jq/ to extract the fields that you need.

You really don't need hadoop/spark/etc for this.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Separate packages per windows subarch in repository

[Jeroen Ooms]

On Thu, Jul 20, 2017 at 1:55 PM, Iago Mosqueira
 wrote:
>
> I maintain a repository of R packages, where some of them contain
> executable binaries. I need to separate those compiled for 32 and 64 bit in
> Windows.

Have a look at the antiword package. It has a simple makevars which
builds antiword$(WIN) executable which is just the 'antiword' on unix,
and antiword32.exe + antiword64.exe on multiarch windows.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Fortran programs not writing stdout on windows

[Jeroen Ooms]

A user has reported an issue that appears when a fortran executable is
called via R on Windows. I am unsure if this expected behavior or a
bug in Fortran or in how R calls Windows executables.

The problem is that when the fortran program is called from R, stdout
gets written to a file "fort.6" instead of stdout. When the same
executable is called from the terminal it works fine and prints to
stdout. This unexpected behavior is unfortunate for R wrappers that
rely on captured output.

A minimal example is available from github [1] and can be installed with

   devtools::install_github("jeroen/ftest")
   ftest::hello()

When running ftest::hello() on linux, R will properly capture output.
However on Windows it will return an empty string, and a file 'fort.6'
gets created in the working directory instead.

The executables can be found in: system.file("bin", package = "ftest")

Interestingly if we open a command line terminal and run the same
executable it prints output to stdout. So perhaps it has to do with
the way R invokes CreateProcess() on Windows?

[1] https://github.com/jeroen/ftest

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Error messages in replayPlot()

[Jeroen Ooms]

I was wondering if there is something that can be done to improve
error messages when replaying a recorded plot. For example a graphics
device that is too small usually results in a helpful error message:

 png(height = 100)
 plot(1)
 # Error in plot.new() : figure margins too large
 dev.off()

However when this happens when replaying a recorded plot, the error
message is not so helpful.

 myplot <- evaluate::evaluate("plot(1)")[[2]]
 png(height = 100)
 replayPlot(myplot)
 # Error in replayPlot(x) : invalid graphics state
 dev.off()

A more informative error message that hints at what exactly is invalid
about the graphics state would be very helpful in this case.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] tempdir() may be deleted during long-running R session

[Jeroen Ooms]

On Tue, Apr 25, 2017 at 1:00 PM, Martin Maechler
 wrote:
> As I've found it is not at all hard to add an option which
> checks the existence and if the directory is no longer "valid",
> tries to recreate it (and if it fails doing that it calls the
> famous R_Suicide(), as it does when R starts up and tempdir()
> cannot be initialized correctly).

Perhaps this can also fix the problem with mcparallel deleting the
tempdir() when one of its children dies:

  file.exists(tempdir()) #TRUE
  parallel::mcparallel(q('no'))
  file.exists(tempdir()) # FALSE

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] potential bug in attribute handling for externalptr

[Jeroen Ooms]

On Fri, Apr 14, 2017 at 11:27 PM, Patrick Perry  wrote:
> Is the following expected behavior? I would expect that modifying y's 
> attributes should not affect x.

Yes this is expected. Environments and external-ptr objects are never
copied on assignment, hence in your example 'x' and 'y' are the same
object. See e.g. ?reg.finalizer or section 5.13 of "writing r
extensions" on external pointers and weak references.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Hyperbolic tangent different results on Windows and Mac

[Jeroen Ooms]

For future reference:
https://sourceforge.net/p/mingw-w64/mailman/message/35747206/

On Wed, Mar 22, 2017 at 2:12 PM, Jeroen Ooms <jeroeno...@gmail.com> wrote:
> This looks like a bug in mingw-w64 CRT. The problem can be produced
> with C++ without R:
>
>   #include 
>   #include 
>   #include 
>
>   int main(){
> std::cout << std::fixed;
> std::complex z(356, 0);
> std::cout << "tanh" << z << " = " << std::tanh(z)
>  << " (tanh(356) = " << std::tanh(356) << ")\n";
>   }
>
> On OS-X we get:
>
>   tanh(356.00,0.00) = (1.00,-0.00) (tanh(356) = 1.00)
>
> But on Windows we get:
>
>   tanh(356.00,0.00) = (nan,0.00) (tanh(356) = 1.00)
>
> I was also able to reproduce the problem with gcc 6.3 in msys2 so it
> has not been fixed upstream. You should file a bug report for
> mingw-w64.
>
> FWIF, we have run into NaN edge-case bugs before with mingw-w64.
>
>  - 
> https://sourceforge.net/p/mingw-w64/mingw-w64/ci/6617ebd5fc6b790c80071d5b1d950e737fc670e1/
>  - 
> https://github.com/wch/r-source/commit/e9aaf8fdeddf27c2a9078cd214a41475c8ff6f40
>
> I am cc'ing Ray Donnelly who is an expert on mingw-w64.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Hyperbolic tangent different results on Windows and Mac

[Jeroen Ooms]

This looks like a bug in mingw-w64 CRT. The problem can be produced
with C++ without R:

  #include 
  #include 
  #include 

  int main(){
std::cout << std::fixed;
std::complex z(356, 0);
std::cout << "tanh" << z << " = " << std::tanh(z)
 << " (tanh(356) = " << std::tanh(356) << ")\n";
  }

On OS-X we get:

  tanh(356.00,0.00) = (1.00,-0.00) (tanh(356) = 1.00)

But on Windows we get:

  tanh(356.00,0.00) = (nan,0.00) (tanh(356) = 1.00)

I was also able to reproduce the problem with gcc 6.3 in msys2 so it
has not been fixed upstream. You should file a bug report for
mingw-w64.

FWIF, we have run into NaN edge-case bugs before with mingw-w64.

 - 
https://sourceforge.net/p/mingw-w64/mingw-w64/ci/6617ebd5fc6b790c80071d5b1d950e737fc670e1/
 - 
https://github.com/wch/r-source/commit/e9aaf8fdeddf27c2a9078cd214a41475c8ff6f40

I am cc'ing Ray Donnelly who is an expert on mingw-w64.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Experimental CXX_STD problem in R 3.4

[Jeroen Ooms]

On Sun, Mar 19, 2017 at 9:09 PM, Martyn Plummer  wrote:
> I have just added some code to ensure that the compilation fails with an 
> informative error message if a specific C++ standard is requested but the 
> corresponding compiler has not been defined. Please test this.

I can confirm that (at least on Windows) we get a proper error message
for CXX14 and CXX17, eg:

  * installing *source* package 'testcxx' ...
  ** libs
  Error in .shlib_internal(args) :
   C++14 standard requested but CXX1Y is not defined
  * removing 'C:/Program Files/R/R-devel/library/testcxx'

CXX98 still does not work though. It seems SHLIB_CXX98LD is undefined
on Windows?

  * installing *source* package 'testcxx' ...
  ** libs
  c:/Rtools/mingw_64/bin/g++  -std=gnu++98
-I"C:/PROGRA~1/R/R-devel/include" -DNDEBUG
-I"d:/Compiler/gcc-4.9.3/local330/include" -O2 -Wall  -mtune=core2
-c test.cpp -o test.o
  -shared -s -static-libgcc -o testcxx.dll tmp.def test.o
-Ld:/Compiler/gcc-4.9.3/local330/lib/x64
-Ld:/Compiler/gcc-4.9.3/local330/lib -LC:/PROGRA~1/R/R-devel/bin/x64
-lR
  -shared: not found
  no DLL was created
  ERROR: compilation failed for package 'testcxx'

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Experimental CXX_STD problem in R 3.4

[Jeroen Ooms]

On Sun, Mar 19, 2017 at 9:09 PM, Martyn Plummer  wrote:
> I have just added some code to ensure that the compilation fails with an 
> informative error message if a specific C++ standard is requested but the 
> corresponding compiler has not been defined. Please test this.

Are you sure we shouldn't just fall back on a previous standard
instead of failing? For example if the package author has specified a
preference for CXX14 but the compiler only has CXX11, the package
might still build with -std=c++11 (given that C++14 is only a small
extension on the C++11 standard).

The current behavior (in R 3.3) for packages with "CXX_STD=CXX11" is
to fall back on CXX when the compiler does not have CXX1X. Will R-3.4
start failing these packages? This would affect many users on CentOS 6
(gcc 4.4.7).

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Experimental CXX_STD problem in R 3.4

[Jeroen Ooms]

R 3.4 has 'experimental' support for setting CXX_STD to CXX98 / CXX11
/ CXX14 / CXX17.

However on most platforms, the R configuration seems to leave the
CXX1Y and CXX1Z fields blank in "${R_HOME}/etc/Makeconf" (rather than
falling back on default CXX). Therefore specifying e.g CXX_STD= CXX14
will fail build with cryptic errors (due to compiling with CXX="")

I don't think this is intended? Some examples from r-devel on Windows:

CXX11: https://win-builder.r-project.org/R8gg703OQSq5/
CXX98: https://win-builder.r-project.org/mpVfXxk79FaN/
CXX14: https://win-builder.r-project.org/L3BSMgAk4cQ7/
CXX17: https://win-builder.r-project.org/3ETZXrgkg77I/

Similar problems appear on Linux. I think the problem is that Makeconf
contains e.g:

CXX1Z =
CXX1ZFLAGS =
CXX1ZPICFLAGS =
CXX1ZSTD =

When CXX_STD contains any other unsupported value (e.g. CXX24) R
simply falls back on the default CXX configuration. The same should
probably happen for e.g. CXX17 when CXX1Z is unset in Makeconf?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Seeking advice regarding compilation of large libraries using RTools (Windows)

[Jeroen Ooms]

On Mon, Mar 6, 2017 at 6:33 AM, Richard Beare  wrote:
> I am working on the SimpleITK package for R. This is an enormous package
> that is largely automatically generated via a set of swig/json/lua magic,
> and is working well under linux and osx.

Is it available somewhere so we can try it?


> However we're having a lot of trouble with the Windows side. In fact, we are 
> struggling to get the base libraries to build using the RTools 3.4 toolchain, 
> even before the worrying about the R-specific parts.

What build environment do you use? The version of gcc with Rtools
should be ok, but the Rtools build utilities in the "bin" folder (in
particular 'make') are old and a frequent source of problems. However
for building external libs you can use other tools, for example those
from msys2. Just make sure you use gcc/g++ from Rtools.


> The current issue is very long time (possibly infinite) linking of dlls, or
> test executables. I've tried using a FAT32 file system for the build, as
> suggested by some old bug reports, but still have the issue.

On Windows you can avoid the run-time dll mess by building static libs
of external libraries. See rwinlib for examples:
https://github.com/rwinlib


> Any suggestions on where to turn next? Are cross compilers the next step?

Try building with msys2, but make sure to use gcc/g++ from Rtools by
setting the `CC` and `CXX` variables in the configure script. Cross
compiling will make things even more complicated because binaries
might not be compatible if your cross compiler has a different version
of gcc or has been configured for another exception model (seh/drawf).

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Registration of native routines

[Jeroen Ooms]

On Tue, Feb 14, 2017 at 5:25 PM, Prof Brian Ripley
 wrote:
>
> Registration has similar benefits to name spaces in R code:
>
> - it ensures that the routines used by .C, .Call etc are those in your 
> package (without needing a PACKAGE argument).
> - it avoids polluting the search space for native routines with those from 
> your package.
> - it checks the number of arguments passed to .Call/.External, and the number 
> and optionally the type for .C/.Fortran.
> - it finds native routines faster, especially if 10s of name spaces are 
> loaded.

Do these benefits also hold for packages that currently use useDynLib
exclusively in combination symbol names? E.g for the example from WRE:

   useDynLib(foo, myRoutine, myOtherRoutine)

Which is invoked via:

  .Call(myRoutine, x, y)

What ambiguity or pollution is introduced by foo:::myRoutine? Should
manually registering 'myRoutine' in C still be mandatory in this case?

It was nice having the NAMESPACE as the central declaration of
callable C routines. The "R_registerRoutines" system will require
maintaining additional C code which re-declares all callable C
functions from other compilation units. This introduces additional
complexity for package authors and might become a source of bugs when
we forget to update the registrations when C functions have changed.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] parallel and tempdir()

[Jeroen Ooms]

A forked process in mcparallel will wipe the (shared) tempdir when it
quits. I think this is almost always undesirable:

  file.exists(tempdir())
  parallel::mcparallel(q("no"))
  file.exists(tempdir()) # false
  file.create(tempfile()) # error

A simple solution might be to skip over R_CleanTempDir when
R_isForkedChild. Example patch: https://git.io/vDwyf

In addition, I would like to suggest to add an option to mcparallel to
set a custom tempdir in the fork to prevent potential conflicts. I
tried implementing this myself but found that setting R_TempDir is not
allowed by CMD check, and Sys_TempDir (which is what actually gets
wiped) is invisible behind attribute_hidden. So I suppose this may
only be done in base.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] parallel::mc*: Is it possible for a child process to know it is a fork?

[Jeroen Ooms]

On Tue, Jan 24, 2017 at 7:06 PM, Henrik Bengtsson
 wrote:
> When using multicore-forking of the parallel package, is it possible
> for a child process to know that it is a fork?

R internally uses R_isForkedChild to prevent certain operations within
the fork. However I don't think this is exported anywhere. You could
do something like:

  extern Rboolean R_isForkedChild;
  SEXP is_forked(){
return ScalarLogical(R_isForkedChild);
  }

But that won't be allowed on CRAN:

* checking compiled code ... NOTE
  Found non-API call to R: ‘R_isForkedChild’
  Compiled code should not call non-API entry points in R.

Another method would be to look at getppid(2) and getpgid(2) to lookup
the parent-id and group-id of the current process and test if it
matches that of the (parent) R process.

If you are only interested in limiting further parallelization within
the fork, perhaps you can simply use parallel::mcaffinity to restrict
the forked process to a single core.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] How to handle INT8 data

[Jeroen Ooms]

On Fri, Jan 20, 2017 at 6:09 PM, Murray Stokely  wrote:
> The lack of 64 bit integer support causes lots of problems when dealing
> with certain types of data where the loss of precision from coercing to 53
> bits with double is unacceptable.
>
> Two packages were developed to deal with this:  int64 and bit64.

Don't forget packages for large arbitrary large numbers such as Rmpfr
and openssl.

  x <- openssl::bignum("12345678987654321")
  x^10

The risk of storing int64 as a double (e.g. in bit64) is that it might
easily be mistaken for a completely different value via unclass() or
Rf_isNumeric() or so.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [R] Problems when trying to install and load package "rzmq"

[Jeroen Ooms]

For those following this thread: rzmq binary packages for Mac and
Windows are now available from CRAN.


On Tue, Jan 3, 2017 at 5:32 PM, Jeroen Ooms <jeroeno...@gmail.com> wrote:
> On Tue, Jan 3, 2017 at 3:53 PM, Whit Armstrong <armstrong.w...@gmail.com> 
> wrote:
>>
>> I maintian the rzmq project.
>>
>> love to get it running on windows, but zmq doesn't play nicely with R's
>> mingw.
>
> It's fairly easy to link against the libraries from rwinlib:
> https://github.com/rwinlib/zeromq. I'll send you a pull request later
> this week to fix the binary packages for win+mac.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [R] Problems when trying to install and load package "rzmq"

[Jeroen Ooms]

On Tue, Jan 3, 2017 at 3:53 PM, Whit Armstrong  wrote:
>
> I maintian the rzmq project.
>
> love to get it running on windows, but zmq doesn't play nicely with R's
> mingw.

It's fairly easy to link against the libraries from rwinlib:
https://github.com/rwinlib/zeromq. I'll send you a pull request later
this week to fix the binary packages for win+mac.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c

[Jeroen Ooms]

On Tue, Dec 20, 2016 at 7:04 AM, Henrik Bengtsson
 wrote:
> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some
> packages don't unload their DLLs when they being unloaded themselves.

I am surprised by this. Why does R not do this automatically? What is
the case for keeping the DLL loaded after the package has been
unloaded? What happens if you reload another version of the same
package from a different library after unloading?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Check package issue on mavericks

[Jeroen Ooms]

I just checked your package on a mavericks VM and it seems fine, so I
wouldn't worry about it. Probably some configuration issue on the CRAN
server, perhaps xquartz is outdated.

Unfortunately there is no easy way anymore to check on Mavericks.
Travis used to support it, but they have deprecated mavericks (along
with Apple and homebrew). Hopefully CRAN will update to the current
MacOS for the next release of R.




On Fri, Dec 9, 2016 at 3:56 PM, Gionata Bocci  wrote:
>
> Dear List,
>
>   I've realized that the "CRAN Checking Package" system returns some
> Warnings for my package TR8 under "r-release-osx-x86_64-mavericks"; all the
> warnings include the following message "Invalid MIT-MAGIC-COOKIE-1 key"
> which I find a little cryptic.
>   After some web searches, I understood that "MIT-MAGIC-COOKIE" is related
> to the X server and most of the messages reporting issues with that are
> related to R packages having problems with "mavericks" (either NOTES or
> Warnings resulting from the "checking process").
>   Not having a Mac available where I could try to explore this issue (and
> not being familiar with Mac in general), I've set up a Travis CI account in
> order to check whether the package was built and "checked --as-cran" on
> both Linux (my OS) and Mac OSX (Travis is using OS X versions >= 10.10): it
> looks like everything is fine, correctly built and checked.
>   I am thus writing to ask if other developers on the list have found and
> fixed this issue (so that I could fix that as well and tick the checkbox "I
> have fixed all problems shown on the package check page" before submitting
> a new version to CRAN).
>   Best,
>
>
> Gionata.
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Support for signing R packages with GPG

[Jeroen Ooms]

On Tue, Oct 25, 2016 at 7:22 PM, Martyn Plummer  wrote:
> Thanks Jeroen. The R Foundation has recently formed a working group to
> look into package authentication. There are basically two models. One
> is the GPG based model you describe; the other is to use X.509 as
> implemented in the PKI package. It's not yet clear which way to go but
> we are thinking about it.

I look forward to hearing what the working group comes up with. I
suppose if you go with x509, CRAN is going to perform CA duties?

Let me know if I can help with implementation, either via gpg or x509.
I am actively developing the openssl package which includes many more
x509 utilities, supporting all common key types (dsa, rsa, ec),
certificate bundles, ssl, etc. The main difference with PKI is that
openssl uses the native pem/der parsers from libssl which are more
robust and also recognize the less common formats, so that we don't
have to deal with parsing/decoding ASN.1 in R.

I will be happy to adapt/extend it further to fit the needs of the
workgroup and help this move forward.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Support for signing R packages with GPG

[Jeroen Ooms]

I would like to propose adding experimental support for including a
PGP signature in R source packages. This would make it possible to
verify the identity of the package author and integrity of the package
sources.

There are two ways to implement this. Assuming GnuPG is on the PATH,
the CMD build script could call:

  gpg --clearsign MD5 -o MD5.gpg

Alternatively the 'gpg' R package provides a more portable method via
the gpgme C library. This method works on Windows / macOS as well.

  writeLines(gpg::gpg_sign("MD5"), "MD5.gpg")

Attached is an example implementation of the latter (also available at
https://git.io/vPb9G) which has been tested with several versions of
GnuPG. It exposes an optional flag for CMD build, i.e:

  R CMD build somepkg --sign
  R CMD build somepkg --sign=jeroen.o...@stat.ucla.edu

The --sign flag creates a signature for the MD5 file [1] in the source
package and saves it as MD5.gpg (similar to a Debian 'Release.gpg'
file [2]). Obviously the package author or build server needs to have
a suitable private key in the local keyring.


## Signature verification

Once R supports signed packages, we can develop a system to take
advantage of such signatures. The verification itself can easily be
implemented via 'gpg --verify' or via gpg::gpg_verify() and could be
performed without changes in R itself. The difficult part in GPG comes
from defining which peers should be trusted.

But even without a 'web of trust' there are several ways one can
immediately take advantage of signatures. For example, when a
installing a package update or dev-version of a package, we can verify
that the signature of the update matches that of the currently
installed package. This would prevent the type of attacks where an
intermediate party pushes a fake malicious update for a popular R
package via e.g. a hacked CRAN mirror.

Eventually, CRAN could consider allowing signatures as a secure
alternative to confirmation emails, and signing packages on the build
servers with a CRAN GPG key, similar to Debian repositories. For now,
at least establishing a format for (optionally) signing packages would
be a great first step.


[1] Eventually we should add SHA256 and SHA256.sig in addition to MD5
[2] 
https://cran.r-project.org/web/packages/gpg/vignettes/intro.html#debian_example
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Cluster: Various GCC, how important is consistency?

[Jeroen Ooms]

On Tue, Oct 18, 2016 at 1:44 AM, Paul Johnson  wrote:
>
> Administrator suggested I try to build with the GCC that is provided
> with the nodes, which is gcc-4.4.7.

Redhat provides an alternative compiler (gcc 5.3 based) in one of it's
opt-in repositories called "redhat developer toolkit" (RDT). In CentOS
you install it as follows:

  yum install -y centos-release-scl
  yum install -y devtoolset-4-gcc-c++

This compiler is specifically designed to be used alongside the EL6
stock gcc 4.4.7. It includes a simple 'enable' script which will put
RDT gcc and g++ in front of your PATH and LD_LIBRARY_PATH and so on.

So what I do on CentOS is install R from EPEL (built with stock gcc
4.4.7) and whenever I need to install an R package that uses e.g.
CXX11, simply start an R shell using the RDT compilers:

   source /opt/rh/devtoolset-4/enable
   R

>From what I have been able to test, this works pretty well (though I
am not a regular EL user). But I was able to build R packages that use
C++11 (such as feather) and once installed, these packages can be used
even in a regular R session (without RDT enabled).

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] package installation fails when symlink of same name exists

[Jeroen Ooms]

On Tue, Jul 19, 2016 at 6:46 PM, Kevin Ushey  wrote:
> R fails to install a package from source over a pre-existing package
> when the path to that package is a symlink, rather than a directory.
> ...
> I don't think anyone's reported this being an issue before

I ran into this as well a while back:
https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16725

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Nested tracing with custom callback

[Jeroen Ooms]

On Wed, Jul 13, 2016 at 5:20 AM, Evan James Patterson
 wrote:
>
> I would like to install a trace function that gets executed whenever *any* R 
> function is called. In Python, for example, this functionality is provided by 
> the `sys.settrace` function.

Maybe you can adapt from the covr package:
https://github.com/jimhester/covr/blob/master/vignettes/how_it_works.Rmd

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Detecting user interrupts in R_tryEval

[Jeroen Ooms]

Is there any way to distinguish between an error and a user
interruption in R_tryEval? In both cases the ErrorOccurred argument is
set to 1. For my application I need a different action in case of a
SIGINT.

>From the source code I infer that R_tryEval basically wraps eval in
R_ToplevelExec, which returns TRUE if fun returns normally, FALSE if
it results in a jump to top level. However both an error and SIGINT
result in a jump. Is there an alternative method, or some method of
finding out which is the two happened after calling R_tryEval?

One thing I tried is see if R_curErrorBuf() is empty. However this is
unreliable because in the case of an interrupt, the error buffer
sometimes contains some old error message.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Suggestion: mkString(NULL) should be NA

[Jeroen Ooms]

On Wed, May 25, 2016 at 12:31 PM, Martin Maechler
 wrote:
> Better than segfaulting, yes, but really agree with Bill (and
> Gabe), also for Rf_mkChar(NULL):
> I think both functions should give an error in such a case
> rather than returning NA_character_
>
> It is an accident of some kind if they got NULL, no?

Not necessarily. A char* of NULL can be a string which is not
initiated or simply unavailable due to configuration.

The example from my original email was in curl package which exposes
the version string of libz that was used to build libcurl:

  mkString(data->libz_version)

This worked on all platforms that I tested. However a user found that
if libcurl was configured --without-libz (which is uncommon) the
libz_version string does not get set by libcurl and is always NULL. I
had not foreseen this and it would lead to a segfault.

I think making mkString() return NA for null strings lead to the most
robust behavior. Raising an exception seems a little harsh to me, as
there is no way the user would be able to recover from this, and there
might not be an actual problem at all.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Suggestion: mkString(NULL) should be NA

[Jeroen Ooms]

On Tue, May 24, 2016 at 5:59 PM, Gabriel Becker  wrote:
> Shouldn't Rf_mkString(NULL) return (the c-level equivalent of) character()
> rather than the NA_character_?

No. It should still be safe to assume that mkString() always returns a
character vector of exactly length one. Anything else could lead to
type errors.

> An empty string and NULL aren't the same.

Exactly! So if you pass in an empty C string, you get an empty R
string, and if you pass in a null pointer you get NA.

Rf_mkString(NULL)   <--> NA
Rf_mkString("")   <--> ""

There is no ambiguity, and much better than segfaulting.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel