Re: [R-SIG-Mac] GTK+ support (or rather lack thereof)

[Tom Elliott]

Simon,

> Hence this is a call to the R community to see if anyone actually cares.
I (and Chris Wild and quite a few of our mac users) care and would greatly 
appreciate working GTK+ CRAN packages! 

I don't have any knowledge re source etc, but just to remind you that the 
current RGtk2 package on CRAN for 3.3 doesn't work (or at least wasn't when I 
last tried): 

library(RGtk2)
gtkHScaleNewWithRange(0, 5, 1)  # segfault


Cheers,

- Tom
___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

Re: [R-SIG-Mac] OpenMP on CRAN

[Simon Urbanek]

Patrick,


> On 4/04/2020, at 1:33 AM, Patrick Schratz  wrote:
> 
> Simon,
> 
> thanks. I assume that you mean “clang from homebrew” = llvm (from homebrew)?

LLVM is the compiler infrastructure project, clang is the C/C++ compiler from 
that project. We don't use any of the other compilers form LLVM.


> Indeed I am currently trying that out and it looks really robust for source 
> installations (more than the systems clang (+ eventual openMP flags like 
> suggested by Kevin) and other variants (gcc from homebrew or openMP enabled 
> clang7 or 8).
> 
> Note that I do not use R via homebrew.
> 
> However, all in all I am essentially mixing the custom llvm from homebrew 
> with the official R installer and the old 10.13 SDK.
> It looks quite complex and I’d wish everything would be easier. However, in 
> the end I just want to have a stable “install from source” environment that 
> works for all packages out there (I do not use the binaries).
> 

Why don't use use Homebrew? That's exactly what Homebrew provides - its own 
ecosystem, everything form source (with cached binaries if you want them).


> All in all I am a bit confused now about all the mixing and options 
> available. Let’s see what the foreseeable future brings and where things are 
> going.
> 

Mixing is not available. You pick one system and go with it, but can't mix 
between them, because of the different run-time libraries.


> Regarding the SDK issue: I think most users just use the binaries on macOS 
> (across all OS versions) and rarely face such issues. And there are 
> presumably not many people out there that do actual R-devel testing on 
> Catalina (?), otherwise I’d expect way more people to jump into the 
> discussion. However, I am not alone with these issues as you can see in the 
> Rcpp issue we were talking about. 
> Maybe you can even reproduce the issues by linking a custom install of the 
> 10.15 SDK on a non-Catalina machine? I don’t know how much trouble that would 
> bring up when trying to jump into the future - but this ist just an idea :)
> 

That's an entirely different, unrelated topic. Testing cutting edge (or even 
pre-release) systems makes sense, but that's not our current worry (there was a 
separate thread about CI).

Cheers,
Simon


> Thanks for your work!
> On 3. Apr 2020, 14:13 +0200, Simon Urbanek , 
> wrote:
>> Patrick,
>> 
>> you were commenting on the thread where we talked about CRAN R - that one is 
>> now compiled using Apple clang. You were talking about using clang from 
>> Homebrew - those are incompatible as they use different run-time. 
>> Unfortunately, the Intel OpenMP run-time varies by clang compiler version 
>> and is known to fail when used with the wrong compiler. Analogously, mixing 
>> gomp (from gcc) and iomp is problematic (and GNU Fortran uses gomp). As 
>> discussed in the Homebrew thread, if you are compiling everything via 
>> Homebrew including R then it's all good, you just can't mix Apple tools and 
>> Homebrew in general.
>> 
>> Also at the bottom I was only talking about 10.14 SDK - that's what we use 
>> on CRAN and I have not seen any issues with it - you were claiming that it 
>> doesn't work with Rcpp and igraph.
>> 
>> Cheers,
>> Simon
>> 
>> 
>> 
>>> On 4/04/2020, at 1:00 AM, Patrick Schratz  wrote:
>>> 
>>> Simon,
>>> 
>>> I don’t understand fully. I am using llvm for all C variants (just now 
>>> shown) in combination with the 10.13 SDK.
>>> So far this combination works flawlessly for all “problematic” packages 
>>> like data.table, igraph or Rcpp.
>>> 
>>> I don’t have deeper knowledge about the “iomp” setup but as of right now I 
>>> don’t know what I am mixing up here. Can you shine more light on this?
>>> If it is about llvm in general: data.table recommends to use llvm openly in 
>>> their wiki due to the lack of openMP for the standard clang.
>>> And while this could be handled as “any other package” in R, we all know 
>>> that it has quite some impact and probably devs who know what they do in 
>>> terms of C configs.
>>> 
>>> Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can’t be 
>>> installed from source with SDK 10.15 and standard apple clang (or any C 
>>> compiler). Switching to SDK 10.13 solves the issue. You were even 
>>> commenting on the Rcpp issue.
>>> Since SDK 10.15 is the current release and many users have no choice on 
>>> which OS version they are (for different reasons), this issue should imo be 
>>> inspected in more detail. What do you think?
>>> 
>>> Best, Patrick
>>> On 2. Apr 2020, 23:38 +0200, Simon Urbanek , 
>>> wrote:
 Patrick,
 
 you can't mix compilers - it really matters which iomp your'e using. llvm 
 has its own modified version which may not be the same as the official 
 Intel release. From your tests it looks like you're using the llvm one 
 which will likely work only with that compiler. Since Apple doesn't have 
 official omp support it's hard to say which versions work and which don't

Re: [R-SIG-Mac] Please test R 4.0.0 pre-releases!

[Berend Hasselman]

Simon,

I have tested an R-4.0.0 (i.e. R version 4.0.0 alpha (2020-04-01 r78130)
with my two packages (nleqslv and geigen) and all my private tests.
I have not found any problems with checking and testing these packages.
(I used Apple clang and Coudert's gfortran 8.2)

The only issue I have encountered is:

The R GUI hangs when I click "What's New In This Version" in the Help menu item.

Berend

> sessionInfo()
R version 4.0.0 alpha (2020-04-01 r78130)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.4

Matrix products: default
BLAS:   
/Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: 
/Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base 

loaded via a namespace (and not attached):
[1] compiler_4.0.0


> On 1 Apr 2020, at 06:27, Simon Urbanek  wrote:
> 
> Dear Mac users,
> 
> R 4.0.0 will be using an entirely new toolchain, entirely new build system on 
> entirely new macOS version and hardware. Therefore I would like to ask you 
> kindly to test the binaries from
> 
> https://mac.R-project.org
> 
> before the release as much as you can. Raising any issues after the release 
> is too late! So please, please, test the pre-releases. Report any issues 
> either directly to me or this mailing list.
> 
> The nightly builds are signed, but not necessarily notarized. However, the 
> build fulfils Apple's conditions and is known to pass notarization (in fact 
> the the package available for download today is actually notarized) so it 
> should be a good test for the release which will be notarized and should work 
> on Catalina.
> 
> For those that want to replicate our setup - technical details: we are now 
> building with macOS 10.13 (High Sierra) as target (i.e. the oldest supported 
> system), regular Apple Xcode/command line tools and GNU Fortran 8.2. R builds 
> are running on macOS 10.15 (Catalina) with Xcode 11.4 using macOS 10.13 
> target. Packages are built on macOS 10.13 VMs with just Apple command line 
> tools (this should make it easy to replicate the setup using Travis, for 
> example). All 3rd party libraries that CRAN uses are available in 
> http://mac.r-project.org/libs-4/
> 
> The new R build system is in
> https://svn.r-project.org/R-dev-web/trunk/QA/Simon/R4
> Packages build system has not changed and is in
> https://svn.r-project.org/R-dev-web/trunk/QA/Simon/packages
> 
> We also plan to have a mac-builder available with similar function as the 
> win-builder where pre-submission tests can be performed and potentially a 
> Travis template.
> 
> Please test R pre-releases and provide feedback!
> 
> Thanks,
> Simon
> 
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

Re: [R-SIG-Mac] GTK+ support (or rather lack thereof)

[Kevin Ushey]

For what it's worth, Homebrew and macports both have scripts for
installing GTK+ from sources, so presumedly those could be cribbed
into a standalone shell script for a build if desired:

https://github.com/Homebrew/homebrew-core/blob/master/Formula/gtk+.rb
https://github.com/macports/macports-ports/blob/master/gnome/gtk2/Portfile

Would that be worth exploring? I'd be willing to try and put something
together if so.

Also -- is the intention to stick with the latest patch release of the
GTK+ 2.x series, or try to upgrade to GTK+ 3.x? (I suspect that
GTK+-using packages would likely need to adapt to changes in GTK+ 3.x)

Kevin

On Thu, Apr 2, 2020 at 6:03 PM Simon Urbanek
 wrote:
>
> We have a fairly complete coverage of packages for R 4.0.0, but one exception 
> is GTK+ (and thus RGtk2 and its dependencies). It seems that GTK+ has been 
> abandoned several years ago, the documented macOS build doesn't work and 
> there are no released binaries. To make things worse, Gnome has been 
> switching from autoconf to custom build systems that are also broken (quite 
> amazing - the build fails with an error in the build system's headers 
> including Python headers...), so the path of building our own release from 
> scratch is also not realistic anymore (we used to build GTK+ for X11 when it 
> was still possible).
>
> Hence this is a call to the R community to see if anyone actually cares. And 
> if so, if there is any known source or path to macOS binaries (script to 
> build it is fine, too). Unlike regular rules, we would allow dynamic linking 
> as we have granted that exception to GTK+ before, but it has to be compatible 
> with the native system. As a last resort, we could also re-use out GTK+ 
> 2.24.17 binaries from Snow Leopard, but those are considerably old, so I'd 
> prefer not to do that. Comments are welcome.
>
> Thanks,
> Simon
>
> ___
> R-SIG-Mac mailing list
> R-SIG-Mac@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac

___
R-SIG-Mac mailing list
R-SIG-Mac@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-mac

Re: [R-SIG-Mac] Homebrew [was: from Mac to LInux?]

[Rainer Krug]

> On 3 Apr 2020, at 13:25, Simon Urbanek  wrote:
> 
> Rainer,
> 
> 
>> On 3/04/2020, at 10:01 PM, Rainer M Krug  wrote:
>> 
>> Thanks a lot for your detailed response - it is very much appreciated.
>> 
>>> On 2 Apr 2020, at 22:30, Simon Urbanek  wrote:
>>> 
>>> 
>>> 
 On 2/04/2020, at 9:15 PM, Patrick Schratz  
 wrote:
 
 AFAIK most people on that list would vote hard against installing R via 
 homebrew for several reasons - maybe there should be a section about this 
 on the R dev / CRAN page to address this topic, @Simon? Otherwise this 
 will come up again and again.
 
>>> 
>> 
>> 
>>> That main objection is that people are mixing Homebrew with CRAN and 
>>> vice-versa which leads to many problems. You cannot use packages from CRAN 
>>> when using Homebrew, period, and you have to be really careful if you want 
>>> to mix Homebrew libraries (not anything R-related) with CRAN (it is doable 
>>> if you know what you're doing).
>> 
>> 
>> This is definitely true (and I assume, you are referring to CRAN binary 
>> packages).
>> 
>> But I am wondering: I am always installing from the normal CRAN mirrors, and 
>> my R installation (homebrew) always installs from source. I never use `type 
>> = “source”`in the installation for packages. 
>> 
>> My question: when would I install incompatible packages (I assume, you mean 
>> binary packages for Mac) from CRAN?
>> 
> 
> Maybe you wouldn't but I have seen people do it.

My question would be: how? Is the installation from binary forced when 
installing with that option?

> 
> 
>> 
>>> 
>>> The fundamental issue is that package managers like Homebrew replace system 
>>> libraries with their own (for new features/updates that the system 
>>> libraries don't provide) which break anything that relies on the system 
>>> library. Out of all the package managers Homebrew the only one that is 
>>> trying to address the issue by trying to separate them, but even that has 
>>> been diverging over time.
>>> 
>>> The second issue is that they are increasingly replacing toolchains 
>>> (compilers) with their own builds, which makes everything incompatible in 
>>> explosive ways (things just segfault). Making sure that a compiler 
>>> toolchain is stable is actually non-trivial and many packager manager 
>>> authors/maintainers have little experience with this. That used the be the 
>>> primary reason to avoid them, because it was just normal that the released 
>>> binaries were miscompiled and things would break all the time. Fortunately, 
>>> I think that has gotten better over time.
>>> 
>>> If you stick only with Homebrew, then you're likely ok, but you're on your 
>>> own and have to compile all packages form sources.
>> 
>> This is true, but I did not encounter big problems. And this is, why I 
>> thought loud in suggesting to setup a (non official!) homebrew tap to 
>> install the packages, similar to how it is in linux (Debian).
>> 
>>> Majority of our time as CRAN maintainers for the binary releases is about 
>>> providing dependent libraries for packages and making sure things "just 
>>> work”.
>> 
>> And I think everybody really appreciates this and is VErY happy about it. 
>> This is essential work!
>> 
>>> It is doable, just a lot of work, so by using Homebrew every user has to 
>>> spend that time.
>> 
>> Doable: definitely. lot of work: in my experience not that much more.
>> 
>> Overall, I definitely agree, that the official distribution should stay the 
>> proper apple way. But there are circumstances, where these are delayed, or 
>> problems occur after macOS upgrades, which will delay the compilation of new 
>> ones (GRASS did not have Mac binaries for a long time due to this problem). 
>> 
> 
> I'm not sure what you refer to, but if there are issues, please let me know. 
> If you don't report issues, they won't get fixed.


No problems here - this was just an example, where a second "approved” 
(whatever that means) installation approach would be useful.


> 
> 
>> So an alternative way of installing, community supported but approved, would 
>> be preferable. The r formula for homebrew has been installed in the last 
>> year more that 112.000 times (https://formulae.brew.sh/formula/r 
>> ) so I think there is likely a rather 
>> large user base of R from homebrew. 
>> 
>> A first step in this direction, would be to either include homebrew in this 
>> list, or create a new R-sig-hoebrew list, which could become the home of 
>> these discussions.
>> 
> 
> I'm not sure I'd like to go there. I would direct that to Homebrew as we have 
> nothing to do with the formulae used there. However, that's not really my 
> call.


At homebrew, you have to get hold of the maintainer of that formula - which is 
often not that easy. Power user problems of a software can only be solved by a 
power user of that software - and writing a formula for homebrew, only requires 
a basic understanding of the program 

Re: [R-SIG-Mac] OpenMP on CRAN

[Patrick Schratz]

Simon,

thanks. I assume that you mean “clang from homebrew” = llvm (from homebrew)?
Indeed I am currently trying that out and it looks really robust for source 
installations (more than the systems clang (+ eventual openMP flags like 
suggested by Kevin) and other variants (gcc from homebrew or openMP enabled 
clang7 or 8).

Note that I do not use R via homebrew.

However, all in all I am essentially mixing the custom llvm from homebrew with 
the official R installer and the old 10.13 SDK.
It looks quite complex and I’d wish everything would be easier. However, in the 
end I just want to have a stable “install from source” environment that works 
for all packages out there (I do not use the binaries).

All in all I am a bit confused now about all the mixing and options available. 
Let’s see what the foreseeable future brings and where things are going.

Regarding the SDK issue: I think most users just use the binaries on macOS 
(across all OS versions) and rarely face such issues. And there are presumably 
not many people out there that do actual R-devel testing on Catalina (?), 
otherwise I’d expect way more people to jump into the discussion. However, I am 
not alone with these issues as you can see in the Rcpp issue we were talking 
about.
Maybe you can even reproduce the issues by linking a custom install of the 
10.15 SDK on a non-Catalina machine? I don’t know how much trouble that would 
bring up when trying to jump into the future - but this ist just an idea :)

Thanks for your work!
On 3. Apr 2020, 14:13 +0200, Simon Urbanek , wrote:
> Patrick,
>
> you were commenting on the thread where we talked about CRAN R - that one is 
> now compiled using Apple clang. You were talking about using clang from 
> Homebrew - those are incompatible as they use different run-time. 
> Unfortunately, the Intel OpenMP run-time varies by clang compiler version and 
> is known to fail when used with the wrong compiler. Analogously, mixing gomp 
> (from gcc) and iomp is problematic (and GNU Fortran uses gomp). As discussed 
> in the Homebrew thread, if you are compiling everything via Homebrew 
> including R then it's all good, you just can't mix Apple tools and Homebrew 
> in general.
>
> Also at the bottom I was only talking about 10.14 SDK - that's what we use on 
> CRAN and I have not seen any issues with it - you were claiming that it 
> doesn't work with Rcpp and igraph.
>
> Cheers,
> Simon
>
>
>
> > On 4/04/2020, at 1:00 AM, Patrick Schratz  wrote:
> >
> > Simon,
> >
> > I don’t understand fully. I am using llvm for all C variants (just now 
> > shown) in combination with the 10.13 SDK.
> > So far this combination works flawlessly for all “problematic” packages 
> > like data.table, igraph or Rcpp.
> >
> > I don’t have deeper knowledge about the “iomp” setup but as of right now I 
> > don’t know what I am mixing up here. Can you shine more light on this?
> > If it is about llvm in general: data.table recommends to use llvm openly in 
> > their wiki due to the lack of openMP for the standard clang.
> > And while this could be handled as “any other package” in R, we all know 
> > that it has quite some impact and probably devs who know what they do in 
> > terms of C configs.
> >
> > Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can’t be 
> > installed from source with SDK 10.15 and standard apple clang (or any C 
> > compiler). Switching to SDK 10.13 solves the issue. You were even 
> > commenting on the Rcpp issue.
> > Since SDK 10.15 is the current release and many users have no choice on 
> > which OS version they are (for different reasons), this issue should imo be 
> > inspected in more detail. What do you think?
> >
> > Best, Patrick
> > On 2. Apr 2020, 23:38 +0200, Simon Urbanek , 
> > wrote:
> > > Patrick,
> > >
> > > you can't mix compilers - it really matters which iomp your'e using. llvm 
> > > has its own modified version which may not be the same as the official 
> > > Intel release. From your tests it looks like you're using the llvm one 
> > > which will likely work only with that compiler. Since Apple doesn't have 
> > > official omp support it's hard to say which versions work and which don't.
> > >
> > >
> > > > On 3/04/2020, at 10:20 AM, Patrick Schratz  
> > > > wrote:
> > > >
> > > > Thanks Kevin, interesting approach.
> > > >
> > > > However, when testing it against a few packages I get symbol pointer 
> > > > issues (e.g. for data.table and xgboost).
> > > > Using llvm everything works.
> > > > Could llvm be the best middle way? Easy installation via brew and still 
> > > > clang compliant.
> > > >
> > > > Currently my Makevars look as follows
> > > >
> > > > CFLAGS=-isysroot 
> > > > /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> > > > […]
> > > >
> > > > CC = ccache /usr/local/opt/llvm/bin/clang
> > > > […]
> > > >
> > > > Where llvm was installed via `brew install llvm`.
> > > > SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 
> > > > 10.

Re: [R-SIG-Mac] OpenMP on CRAN

[Simon Urbanek]

Patrick,

you were commenting on the thread where we talked about CRAN R - that one is 
now compiled using Apple clang. You were talking about using clang from 
Homebrew - those are incompatible as they use different run-time. 
Unfortunately, the Intel OpenMP run-time varies by clang compiler version and 
is known to fail when used with the wrong compiler. Analogously, mixing gomp 
(from gcc) and iomp is problematic (and GNU Fortran uses gomp). As discussed in 
the Homebrew thread, if you are compiling everything via Homebrew including R 
then it's all good, you just can't mix Apple tools and Homebrew in general.

Also at the bottom I was only talking about 10.14 SDK - that's what we use on 
CRAN and I have not seen any issues with it - you were claiming that it doesn't 
work with Rcpp and igraph.

Cheers,
Simon



> On 4/04/2020, at 1:00 AM, Patrick Schratz  wrote:
> 
> Simon,
> 
> I don’t understand fully. I am using llvm for all C variants (just now shown) 
> in combination with the 10.13 SDK.
> So far this combination works flawlessly for all “problematic” packages like 
> data.table, igraph or Rcpp.
> 
> I don’t have deeper knowledge about the “iomp” setup but as of right now I 
> don’t know what I am mixing up here. Can you shine more light on this?
> If it is about llvm in general: data.table recommends to use llvm openly in 
> their wiki due to the lack of openMP for the standard clang.
> And while this could be handled as “any other package” in R, we all know that 
> it has quite some impact and probably devs who know what they do in terms of 
> C configs.
> 
> Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can’t be installed 
> from source with SDK 10.15 and standard apple clang (or any C compiler). 
> Switching to SDK 10.13 solves the issue. You were even commenting on the Rcpp 
> issue.
> Since SDK 10.15 is the current release and many users have no choice on which 
> OS version they are (for different reasons), this issue should imo be 
> inspected in more detail. What do you think?
> 
> Best, Patrick
> On 2. Apr 2020, 23:38 +0200, Simon Urbanek , 
> wrote:
>> Patrick,
>> 
>> you can't mix compilers - it really matters which iomp your'e using. llvm 
>> has its own modified version which may not be the same as the official Intel 
>> release. From your tests it looks like you're using the llvm one which will 
>> likely work only with that compiler. Since Apple doesn't have official omp 
>> support it's hard to say which versions work and which don't.
>> 
>> 
>>> On 3/04/2020, at 10:20 AM, Patrick Schratz  
>>> wrote:
>>> 
>>> Thanks Kevin, interesting approach.
>>> 
>>> However, when testing it against a few packages I get symbol pointer issues 
>>> (e.g. for data.table and xgboost).
>>> Using llvm everything works.
>>> Could llvm be the best middle way? Easy installation via brew and still 
>>> clang compliant.
>>> 
>>> Currently my Makevars look as follows
>>> 
>>> CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
>>> […]
>>> 
>>> CC = ccache /usr/local/opt/llvm/bin/clang
>>> […]
>>> 
>>> Where llvm was installed via `brew install llvm`.
>>> SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 10.15
>> 
>> 
>> I mentioned that before, but I do not see issues with 10.14 SDK.
>> 
>> Cheers,
>> Simon
>> 
>> 
>>> On 2. Apr 2020, 23:01 +0200, Simon Urbanek , 
>>> wrote:
 Tim,
 
 
> On 3/04/2020, at 2:01 AM, BATES Timothy  wrote:
> 
> Moving to a compiler that drops support for OpenMP seems a sad choice, 
> especially now we’ve all climbed the learning curve of the non-Apple 
> compiler (the real barrier was lack of a pkg installer and that’s done 
> now).
> 
 
 It has caused a lot of issues, it trips people on a daily basis which is 
 just not worth it. Also with Apple's increasingly strict rules about what 
 can be distributed we don't want to be in the business in maintaining a 
 separate toolchain. There have been numerous issues with C++ exceptions so 
 the fall out was much bigger than originally expected and it would only 
 get worse.
 
 
> Losing OpenMP for the CRAN version of OpenMx/umx (our SEM packages) would 
> be a big loss for users (for whom the CRAN version now supports OpenMP 
> giving them a 2-12x speed up). In general, R on Mac is made more viable 
> by having OpenMP
> 
> Re Brian’s points, I’d say that the distribution problem is crucial: 
> Packages not on CRAN have dramatically diminished accessibility/useage.
> 
 
 The idea is that if a package deems this critical, it can enable this for 
 the users. As it is now, packages can still supply iomp and use it.
 
 That said, I would open for discussion the ability to distribute iomp with 
 the R binary, but it would not be supported by R directly, i.e., it would 
 be on the package author to make sure that the way the package operates is 
 c

Re: [R-SIG-Mac] OpenMP on CRAN

[Patrick Schratz]

Simon,

I don’t understand fully. I am using llvm for all C variants (just now shown) 
in combination with the 10.13 SDK.
So far this combination works flawlessly for all “problematic” packages like 
data.table, igraph or Rcpp.

I don’t have deeper knowledge about the “iomp” setup but as of right now I 
don’t know what I am mixing up here. Can you shine more light on this?
If it is about llvm in general: data.table recommends to use llvm openly in 
their wiki due to the lack of openMP for the standard clang.
And while this could be handled as “any other package” in R, we all know that 
it has quite some impact and probably devs who know what they do in terms of C 
configs.

Regarding the SDK issue: Both Rcpp v1.0.4 and igraph 1.2.5 can’t be installed 
from source with SDK 10.15 and standard apple clang (or any C compiler). 
Switching to SDK 10.13 solves the issue. You were even commenting on the Rcpp 
issue.
Since SDK 10.15 is the current release and many users have no choice on which 
OS version they are (for different reasons), this issue should imo be inspected 
in more detail. What do you think?

Best, Patrick
On 2. Apr 2020, 23:38 +0200, Simon Urbanek , wrote:
> Patrick,
>
> you can't mix compilers - it really matters which iomp your'e using. llvm has 
> its own modified version which may not be the same as the official Intel 
> release. From your tests it looks like you're using the llvm one which will 
> likely work only with that compiler. Since Apple doesn't have official omp 
> support it's hard to say which versions work and which don't.
>
>
> > On 3/04/2020, at 10:20 AM, Patrick Schratz  
> > wrote:
> >
> > Thanks Kevin, interesting approach.
> >
> > However, when testing it against a few packages I get symbol pointer issues 
> > (e.g. for data.table and xgboost).
> > Using llvm everything works.
> > Could llvm be the best middle way? Easy installation via brew and still 
> > clang compliant.
> >
> > Currently my Makevars look as follows
> >
> > CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
> > […]
> >
> > CC = ccache /usr/local/opt/llvm/bin/clang
> > […]
> >
> > Where llvm was installed via `brew install llvm`.
> > SDK 10.13 because of {igraph} and {Rcpp} issues with SDK 10.14 and SDK 10.15
>
>
> I mentioned that before, but I do not see issues with 10.14 SDK.
>
> Cheers,
> Simon
>
>
> > On 2. Apr 2020, 23:01 +0200, Simon Urbanek , 
> > wrote:
> > > Tim,
> > >
> > >
> > > > On 3/04/2020, at 2:01 AM, BATES Timothy  wrote:
> > > >
> > > > Moving to a compiler that drops support for OpenMP seems a sad choice, 
> > > > especially now we’ve all climbed the learning curve of the non-Apple 
> > > > compiler (the real barrier was lack of a pkg installer and that’s done 
> > > > now).
> > > >
> > >
> > > It has caused a lot of issues, it trips people on a daily basis which is 
> > > just not worth it. Also with Apple's increasingly strict rules about what 
> > > can be distributed we don't want to be in the business in maintaining a 
> > > separate toolchain. There have been numerous issues with C++ exceptions 
> > > so the fall out was much bigger than originally expected and it would 
> > > only get worse.
> > >
> > >
> > > > Losing OpenMP for the CRAN version of OpenMx/umx (our SEM packages) 
> > > > would be a big loss for users (for whom the CRAN version now supports 
> > > > OpenMP giving them a 2-12x speed up). In general, R on Mac is made more 
> > > > viable by having OpenMP
> > > >
> > > > Re Brian’s points, I’d say that the distribution problem is crucial: 
> > > > Packages not on CRAN have dramatically diminished accessibility/useage.
> > > >
> > >
> > > The idea is that if a package deems this critical, it can enable this for 
> > > the users. As it is now, packages can still supply iomp and use it.
> > >
> > > That said, I would open for discussion the ability to distribute iomp 
> > > with the R binary, but it would not be supported by R directly, i.e., it 
> > > would be on the package author to make sure that the way the package 
> > > operates is compatible with that binary. Let me know what you think.
> > >
> > >
> > > > Second, a great range of compute-intensive problems are amenable to 
> > > > division amongst cores, including nearly all models that take more than 
> > > > a nominal amount of time to run: So simulations, CIs, bootstrapping, 
> > > > nearly everything in genetics all speeds up.
> > > >
> > >
> > > Yes, but OpenMP is rarely used for those tasks. In R OpenMP can be only 
> > > used for very small subset of such tasks - which is why alternative 
> > > approaches are much more common.
> > >
> > > Cheers,
> > > Simon
> > >
> > >
> > > > I’d say especially on desktop/laptop. The big advantage of multi blade 
> > > > systems requires snowfall-type solutions, but desktops profit 
> > > > automatically from their multi-core structure and don;’t have multiple 
> > > > processors (except graphics, which no-one seems to be exploiting on 
> > > > CRA

Re: [R-SIG-Mac] Homebrew [was: from Mac to LInux?]

[Simon Urbanek]

Rainer,


> On 3/04/2020, at 10:01 PM, Rainer M Krug  wrote:
> 
> Thanks a lot for your detailed response - it is very much appreciated.
> 
>> On 2 Apr 2020, at 22:30, Simon Urbanek  wrote:
>> 
>> 
>> 
>>> On 2/04/2020, at 9:15 PM, Patrick Schratz  wrote:
>>> 
>>> AFAIK most people on that list would vote hard against installing R via 
>>> homebrew for several reasons - maybe there should be a section about this 
>>> on the R dev / CRAN page to address this topic, @Simon? Otherwise this will 
>>> come up again and again.
>>> 
>> 
> 
> 
>> That main objection is that people are mixing Homebrew with CRAN and 
>> vice-versa which leads to many problems. You cannot use packages from CRAN 
>> when using Homebrew, period, and you have to be really careful if you want 
>> to mix Homebrew libraries (not anything R-related) with CRAN (it is doable 
>> if you know what you're doing).
> 
> 
> This is definitely true (and I assume, you are referring to CRAN binary 
> packages).
> 
> But I am wondering: I am always installing from the normal CRAN mirrors, and 
> my R installation (homebrew) always installs from source. I never use `type = 
> “source”`in the installation for packages. 
> 
> My question: when would I install incompatible packages (I assume, you mean 
> binary packages for Mac) from CRAN?
> 

Maybe you wouldn't but I have seen people do it.


> 
>> 
>> The fundamental issue is that package managers like Homebrew replace system 
>> libraries with their own (for new features/updates that the system libraries 
>> don't provide) which break anything that relies on the system library. Out 
>> of all the package managers Homebrew the only one that is trying to address 
>> the issue by trying to separate them, but even that has been diverging over 
>> time.
>> 
>> The second issue is that they are increasingly replacing toolchains 
>> (compilers) with their own builds, which makes everything incompatible in 
>> explosive ways (things just segfault). Making sure that a compiler toolchain 
>> is stable is actually non-trivial and many packager manager 
>> authors/maintainers have little experience with this. That used the be the 
>> primary reason to avoid them, because it was just normal that the released 
>> binaries were miscompiled and things would break all the time. Fortunately, 
>> I think that has gotten better over time.
>> 
>> If you stick only with Homebrew, then you're likely ok, but you're on your 
>> own and have to compile all packages form sources.
> 
> This is true, but I did not encounter big problems. And this is, why I 
> thought loud in suggesting to setup a (non official!) homebrew tap to install 
> the packages, similar to how it is in linux (Debian).
> 
>> Majority of our time as CRAN maintainers for the binary releases is about 
>> providing dependent libraries for packages and making sure things "just 
>> work”.
> 
> And I think everybody really appreciates this and is VErY happy about it. 
> This is essential work!
> 
>> It is doable, just a lot of work, so by using Homebrew every user has to 
>> spend that time.
> 
> Doable: definitely. lot of work: in my experience not that much more.
> 
> Overall, I definitely agree, that the official distribution should stay the 
> proper apple way. But there are circumstances, where these are delayed, or 
> problems occur after macOS upgrades, which will delay the compilation of new 
> ones (GRASS did not have Mac binaries for a long time due to this problem). 
> 

I'm not sure what you refer to, but if there are issues, please let me know. If 
you don't report issues, they won't get fixed.


> So an alternative way of installing, community supported but approved, would 
> be preferable. The r formula for homebrew has been installed in the last year 
> more that 112.000 times (https://formulae.brew.sh/formula/r) so I think there 
> is likely a rather large user base of R from homebrew. 
> 
> A first step in this direction, would be to either include homebrew in this 
> list, or create a new R-sig-hoebrew list, which could become the home of 
> these discussions.
> 

I'm not sure I'd like to go there. I would direct that to Homebrew as we have 
nothing to do with the formulae used there. However, that's not really my call.

Today another reason came up why I have an issue with Homebrew: people install 
compilers and similar tools and then use them instead of Xcode for everything 
outside of Homebrew. That is absolutely unsupported, because they have 
different runtime environments, so things break. Sometimes in subtle ways and 
things segfault at random points. The thing is, if people knew what they were 
doing, and Homebrew was only used by developers, it would be all fine - they'd 
know not to mix and match run-times. But most user don't.


> 
>> 
>> (FWIW I use Hombrew myself for tools, but not in /usr/local (I'm using 
>> /opt/brew) and I only put it on the PATH for the tools that I need, never to 
>> compile anything "native" against it.)
> 
> T

Re: [R-SIG-Mac] Homebrew [was: from Mac to LInux?]

[Rainer M Krug]

Thanks a lot for your detailed response - it is very much appreciated.

> On 2 Apr 2020, at 22:30, Simon Urbanek  wrote:
> 
> 
> 
>> On 2/04/2020, at 9:15 PM, Patrick Schratz  wrote:
>> 
>> AFAIK most people on that list would vote hard against installing R via 
>> homebrew for several reasons - maybe there should be a section about this on 
>> the R dev / CRAN page to address this topic, @Simon? Otherwise this will 
>> come up again and again.
>> 
> 


> That main objection is that people are mixing Homebrew with CRAN and 
> vice-versa which leads to many problems. You cannot use packages from CRAN 
> when using Homebrew, period, and you have to be really careful if you want to 
> mix Homebrew libraries (not anything R-related) with CRAN (it is doable if 
> you know what you're doing).


This is definitely true (and I assume, you are referring to CRAN binary 
packages).

But I am wondering: I am always installing from the normal CRAN mirrors, and my 
R installation (homebrew) always installs from source. I never use `type = 
“source”`in the installation for packages. 

My question: when would I install incompatible packages (I assume, you mean 
binary packages for Mac) from CRAN?


> 
> The fundamental issue is that package managers like Homebrew replace system 
> libraries with their own (for new features/updates that the system libraries 
> don't provide) which break anything that relies on the system library. Out of 
> all the package managers Homebrew the only one that is trying to address the 
> issue by trying to separate them, but even that has been diverging over time.
> 
> The second issue is that they are increasingly replacing toolchains 
> (compilers) with their own builds, which makes everything incompatible in 
> explosive ways (things just segfault). Making sure that a compiler toolchain 
> is stable is actually non-trivial and many packager manager 
> authors/maintainers have little experience with this. That used the be the 
> primary reason to avoid them, because it was just normal that the released 
> binaries were miscompiled and things would break all the time. Fortunately, I 
> think that has gotten better over time.
> 
> If you stick only with Homebrew, then you're likely ok, but you're on your 
> own and have to compile all packages form sources.

This is true, but I did not encounter big problems. And this is, why I thought 
loud in suggesting to setup a (non official!) homebrew tap to install the 
packages, similar to how it is in linux (Debian).

> Majority of our time as CRAN maintainers for the binary releases is about 
> providing dependent libraries for packages and making sure things "just work”.

And I think everybody really appreciates this and is VErY happy about it. This 
is essential work!

> It is doable, just a lot of work, so by using Homebrew every user has to 
> spend that time.

Doable: definitely. lot of work: in my experience not that much more.

Overall, I definitely agree, that the official distribution should stay the 
proper apple way. But there are circumstances, where these are delayed, or 
problems occur after macOS upgrades, which will delay the compilation of new 
ones (GRASS did not have Mac binaries for a long time due to this problem). 

So an alternative way of installing, community supported but approved, would be 
preferable. The r formula for homebrew has been installed in the last year more 
that 112.000 times (https://formulae.brew.sh/formula/r) so I think there is 
likely a rather large user base of R from homebrew. 

A first step in this direction, would be to either include homebrew in this 
list, or create a new R-sig-hoebrew list, which could become the home of these 
discussions.


> 
> (FWIW I use Hombrew myself for tools, but not in /usr/local (I'm using 
> /opt/brew) and I only put it on the PATH for the tools that I need, never to 
> compile anything "native" against it.)

Than you are completely in the cold and without support from homebrew… As far 
as I heard, in this case, everything is locally compiled and not using the 
binaries - correct?

Cheers, and thanks again for your very useful clarifications,

Rainer

> 
> Cheers,
> Simon
> 
> 
> 
>> Anyhow, this is also not relating to the initial topic of that thread and 
>> should probably discussed separately.
>> On 2. Apr 2020, 10:04 +0200, Rainer M Krug , wrote:
>>> I am using Homebrew on a Mac (two Macs - one at home, one at work) instead 
>>> of the official R package, and I did not have any problems after upgrades - 
>>> maybe I am lucky, maybe not as picky in defining “problem”, but my 
>>> suggestion would be to try R from homebrew to install R.
>>> 
>>> OK - no support from here - I know.
>>> 
>>> And homebrew has also binary versions. What is missing, is a hombrew R 
>>> package repository. Maybe an idea to create one?
>>> 
>>> 
>>> Cheers,
>>> 
>>> Rainer
>>> 
>>> 
 On 2 Apr 2020, at 02:37, Duncan Murdoch  wrote:
 
 On 01/04/2020 2:48 p.m., Carl Witthoft wrote: