Re: [Bioc-devel] fixing deltaCaptureC

[Jennifer Wokaty]

Hi Michael,

I see that it is passing for 3.18. The email might have been for devel, which 
is still failing. You should cherry-pick the commit and bump the version for 
the devel branch: 
https://contributions.bioconductor.org/git-version-control.html?q=cherry%20pick#bug-fix-in-release-and-devel.

Jennifer Wokaty (they/them)

Waldron Lab at CUNY SPH
Bioconductor Core Team

From: Bioc-devel  on behalf of Michael 
Shapiro 
Sent: Wednesday, March 27, 2024 8:06 AM
To: bioc-devel@r-project.org 
Subject: [Bioc-devel] fixing deltaCaptureC

* This email originates from a sender outside of CUNY. Verify the sender before 
replying or clicking on links and attachments. *

My package deltaCaptureC is suddenly having build problems, specifically a 
problem building the vignette.  It fails when attempting to display a ggplot 
object in the vignette, and this seems to be due to changes in ggplot.  I've 
updated the .rda files for the plots and the vignette now builds properly 
locally.  I bumped the version and pushed to RELEASE_3_18.  I did this last 
week, but yesterday I got an email that it is still failing to build.

Please advise.

Many thanks,
Michael


The Francis Crick Institute Limited is a registered charity in England and 
Wales no. 1140062 and a company registered in England and Wales no. 06885462, 
with its registered office at 1 Midland Road London NW1 1AT

[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel__%3B!!PxiZbSOawA!JHWqiJnNK-f8PoNHtJ-M96fMyX_swfUn9wmO5qwqAT_uD8hqueXvqS5MuQKMg4BHDKPR9-zmQUfD9SupcC58xTHtF7uPO5uhQg0Uqvf1%24=05%7C02%7Cjennifer.wokaty%40sph.cuny.edu%7C14abad3238fc483cafc208dc4e577cb1%7C6f60f0b35f064e099715989dba8cc7d8%7C0%7C0%7C638471384882140984%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=z21%2BwXb4PzbWrF4uTeL2WIWcdauXw7Qmd3n7cGSd9d4%3D=0

[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [Rd] paths capability FALSE on devel?

[Trevor Davis]

In the past I've observed similar behaviour with R compiled with support
for cairo but no pango:

https://stat.ethz.ch/pipermail/r-devel/2022-April/081587.html

Despite cairo support if no pango then the documentation in ?X11 says R
defaults to type "Xlib" instead of type "cairo" even though both seem to
work and only type "cairo" supports all the new grid features like paths...

What happens if you set `X11.options(type = "cairo")`?

Trevor

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Bioc-devel] about deprecate package DNABarcode

[Kern, Lori via Bioc-devel]

You'll have to remind them to request un-deprecation here or on the emails they 
were reached out to.  We will not un-deprecate until the package is fixed and 
they request the package not be deprecated to prove they have an active email.


Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263


From: Pedro S�nchez S�nchez 
Sent: Wednesday, March 27, 2024 10:23 AM
To: Sun Wenjie 
Cc: Kern, Lori ; bioc-devel@r-project.org 
; buschm...@b-n-w.org 
Subject: Re: [Bioc-devel] about deprecate package DNABarcode

Hi Sun,

Yes! I was about to write the same to you hahaha

It's great news!

Best,

Pedro


On Wed, Mar 27, 2024, 15:18 Sun Wenjie 
mailto:wenjie@curie.fr>> wrote:
Hi Pedro,

Tilo has responded and is currently updating the DNABarcode package.

Best,
Wenjie


From: Pedro S�nchez S�nchez 
mailto:bio.pedro.technol...@gmail.com>>
Sent: Wednesday, March 20, 2024 1:58 PM
To: Kern, Lori
Cc: Sun Wenjie; bioc-devel@r-project.org
Subject: Re: [Bioc-devel] about deprecate package DNABarcode

WARNING: This email originated from outside of Institut Curie. Please validate 
the sender's email address before clicking on links or attachments as they may 
not be safe.

Hi all,

I also tried to contact the author on the available email contacts (waiting for 
response from 
tilo.buschmann...@gmail.com>;
 depreciated email 
tilo.buschm...@izi.fraunhofer.de>)
 I found on the internet. If I get a response, I'll let you know.

It will be a pity if DNABarcodes is depreciated for lacking of maintenance.

Just in case, it is helpful for someone, I just want to mention there is an 
apparently similar repository on GitHub with the same goal as DNABarcodes, by 
other developers and written in Python/Jupyter Notebooks: 
https://github.com/feldman4/dna-barcodes.

Best,

Pedro

El mar, 12 mar 2024 a las 13:18, Kern, Lori via Bioc-devel 
(mailto:bioc-devel@r-project.org>>>)
 escribi�:
The package is deprecated in devel.  It will remain listed in the next Bioc 
3.19 release, but deprecated, and removed from anticipated 3.20.  The package 
will have up until the fall release (generally late Oct / early Nov) to 
reinstate before it will be removed.

Cheers,


Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263


From: Bioc-devel 
mailto:bioc-devel-boun...@r-project.org>>>
 on behalf of Sun Wenjie 
mailto:wenjie@curie.fr>>>
Sent: Tuesday, March 12, 2024 8:07 AM
To: 
bioc-devel@r-project.org>
 
mailto:bioc-devel@r-project.org>>>
Subject: [Bioc-devel] about deprecate package DNABarcode

Hi Bioconductor core team,

I saw the package of DNABarcode is listed as to be deprecated.

I am trying to reach the author. Do you know when will be the deadline?

Best,
Wenjie

--
Wenjie SUN
Postdoctoral researcher
UMR168, Curie Institute
E-mail: 
wenjie@curie.fr>
Address: batiment Curie, RDC.13A
11 rue Pierre et Marie Curie, 75005 Paris
https://secure-web.cisco.com/1-TBJw4o6EGDXK_14o3XCIp7DQhMLTaxvVR4NCL9q9sGYrycpujC_OYkx7ortrnjf5crYpYXyRsFEcpldkHYYLTfw-V9oZUKv8flhupsI5LV_JzcWwDb1FnvBDTYwcj0cXxmA3GNJQVF3Mnc1QnR55FA-_sO9AKIMHbDHD2U1y9HvFChc3nLydpoM4IZQhFrkC01Z7BtNAdt0gmfJzyF8AM4qlM40FR8g5KAPURr03QM4ukAWrszSyzsi9jEVmrUnL0Tjcsf1thPL1wOBupAU-UZtzsKkLIae_8gglI6k6W8kBMlUjp2SvfS4iEIah835/https%3A%2F%2Fcurie.fr%2Fequipe%2Fperie

___
Bioc-devel@r-project.org>
 mailing list

Re: [Bioc-devel] about deprecate package DNABarcode

[Pedro Sánchez Sánchez]

Hi Sun,

Yes! I was about to write the same to you hahaha

It's great news!

Best,

Pedro


On Wed, Mar 27, 2024, 15:18 Sun Wenjie  wrote:

> Hi Pedro,
>
> Tilo has responded and is currently updating the DNABarcode package.
>
> Best,
> Wenjie
>
> 
> From: Pedro Sánchez Sánchez 
> Sent: Wednesday, March 20, 2024 1:58 PM
> To: Kern, Lori
> Cc: Sun Wenjie; bioc-devel@r-project.org
> Subject: Re: [Bioc-devel] about deprecate package DNABarcode
>
> WARNING: This email originated from outside of Institut Curie. Please
> validate the sender's email address before clicking on links or attachments
> as they may not be safe.
>
> Hi all,
>
> I also tried to contact the author on the available email contacts
> (waiting for response from tilo.buschmann...@gmail.com tilo.buschmann...@gmail.com>; depreciated email
> tilo.buschm...@izi.fraunhofer.de)
> I found on the internet. If I get a response, I'll let you know.
>
> It will be a pity if DNABarcodes is depreciated for lacking of maintenance.
>
> Just in case, it is helpful for someone, I just want to mention there is
> an apparently similar repository on GitHub with the same goal as
> DNABarcodes, by other developers and written in Python/Jupyter Notebooks:
> https://github.com/feldman4/dna-barcodes.
>
> Best,
>
> Pedro
>
> El mar, 12 mar 2024 a las 13:18, Kern, Lori via Bioc-devel (<
> bioc-devel@r-project.org>) escribió:
> The package is deprecated in devel.  It will remain listed in the next
> Bioc 3.19 release, but deprecated, and removed from anticipated 3.20.  The
> package will have up until the fall release (generally late Oct / early
> Nov) to reinstate before it will be removed.
>
> Cheers,
>
>
> Lori Shepherd - Kern
>
> Bioconductor Core Team
>
> Roswell Park Comprehensive Cancer Center
>
> Department of Biostatistics & Bioinformatics
>
> Elm & Carlton Streets
>
> Buffalo, New York 14263
>
> 
> From: Bioc-devel  bioc-devel-boun...@r-project.org>> on behalf of Sun Wenjie <
> wenjie@curie.fr>
> Sent: Tuesday, March 12, 2024 8:07 AM
> To: bioc-devel@r-project.org <
> bioc-devel@r-project.org>
> Subject: [Bioc-devel] about deprecate package DNABarcode
>
> Hi Bioconductor core team,
>
> I saw the package of DNABarcode is listed as to be deprecated.
>
> I am trying to reach the author. Do you know when will be the deadline?
>
> Best,
> Wenjie
>
> --
> Wenjie SUN
> Postdoctoral researcher
> UMR168, Curie Institute
> E-mail: wenjie@curie.fr
> Address: batiment Curie, RDC.13A
> 11 rue Pierre et Marie Curie, 75005 Paris
>
> https://secure-web.cisco.com/1-TBJw4o6EGDXK_14o3XCIp7DQhMLTaxvVR4NCL9q9sGYrycpujC_OYkx7ortrnjf5crYpYXyRsFEcpldkHYYLTfw-V9oZUKv8flhupsI5LV_JzcWwDb1FnvBDTYwcj0cXxmA3GNJQVF3Mnc1QnR55FA-_sO9AKIMHbDHD2U1y9HvFChc3nLydpoM4IZQhFrkC01Z7BtNAdt0gmfJzyF8AM4qlM40FR8g5KAPURr03QM4ukAWrszSyzsi9jEVmrUnL0Tjcsf1thPL1wOBupAU-UZtzsKkLIae_8gglI6k6W8kBMlUjp2SvfS4iEIah835/https%3A%2F%2Fcurie.fr%2Fequipe%2Fperie
>
> ___
> Bioc-devel@r-project.org mailing list
>
> https://secure-web.cisco.com/1l6kli-5ER6TfFRoWON7eg605W9-aPQ1auinsEup1uN5OFQqRFc_QRRlrkmssHncngEABAwXz4PYezr9ontykNdw3OQlPlaKaQJXx9J_ey5JecbnynELh3K-b_GgnpbL94Du22Rl80OCRrkiU3mGuPhIqnUgFxA9Bo9QsZmZHs3u4yuLvZK9QVQxgvloJXJO8Z0H8rBvABpOzApweEY8hv8ovHZBVkWCCPX1r-yGlLt9G_Y1GJvdqruefoEmWpUBxngKD3odX73chnPmSKwS0X31BlZezoSG7BeE5KK_SR8tKGHjf8oxPHoNBTC1WTP6x/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel
>
>
>
> This email message may contain legally privileged and/or confidential
> information.  If you are not the intended recipient(s), or the employee or
> agent responsible for the delivery of this message to the intended
> recipient(s), you are hereby notified that any disclosure, copying,
> distribution, or use of this email message is prohibited.  If you have
> received this message in error, please notify the sender immediately by
> e-mail and delete this email message from your computer. Thank you.
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>

[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [R-pkg-devel] using portable simd instructions

[Serguei Sokol]

Le 27/03/2024 à 14:54, jesse koops a écrit :

I tried that but I found the interface awkward and there was really no
performance bonus. It was in the early phase of experimentation and I
didn't save it, so it could very well be that I got the compiler
settings wrong and simd was not used. But if that was the case, there
would still be the problem of using the correct compiler settings
cross platform.

When I compile the example of the cited page with "authorized" flag "-std":

   g++ -std=c++20 stdx_simd.cpp -o tmp.exe

then I do:

   objdump -d tmp.exe > tmp.asm

I do find simd instructions in assembler code, e.g.:

   grep paddd tmp.asm

14a8:   66 0f fe c1 paddd  %xmm1,%xmm0
8f7b:   66 0f fe c1 paddd  %xmm1,%xmm0



Op wo 27 mrt 2024 om 14:44 schreef Serguei Sokol :


Le 26/03/2024 à 15:51, Tomas Kalibera a écrit :


On 3/26/24 10:53, jesse koops wrote:

Hello R-package-devel,

I recently got inspired by the rcppsimdjson package to try out simd
registers. It works fantastic on my computer but I struggle to find
information on how to make it portable. It doesn't help in this case
that R and Rcpp make including Cpp code so easy that I have never had
to learn about cmake and compiler flags. I would appreciate any help,
including of the type: "go read instructions at ...".

I use RcppArmadillo and Rcpp. I currenlty include the following header:

#include 

The functions in immintrin that I use are:

_mm256_loadu_pd
_mm256_set1_pd
_mm256_mul_pd
_mm256_fmadd_pd
_mm256_storeu_pd

and I define up to four __m256d registers. From information found
online (not sure where anymore) I constructed the following makevars
file:

CXX_STD = CXX14

PKG_CPPFLAGS = -I../inst/include -mfma -msse4.2 -mavx

PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)

(I also use openmp, that has always worked fine, I just included all
lines for completeness) Rcheck gives me two notes:

─  using R version 4.3.2 (2023-10-31 ucrt)
─  using platform: x86_64-w64-mingw32 (64-bit)
─  R was compiled by
 gcc.exe (GCC) 12.3.0
 GNU Fortran (GCC) 12.3.0

❯ checking compilation flags used ... NOTE
Compilation used the following non-portable flag(s):
  '-mavx' '-mfma' '-msse4.2'

❯ checking C++ specification ... NOTE
  Specified C++14: please drop specification unless essential

But as far as I understand, the flags are necessary, at least in GCC.
How can I make this portable and CRAN-acceptable?


I think it the best way for portability is to use a higher-level library
that already has done the low-level business of maintaining multiple
versions of the code (with multiple instruction sets) and choosing one
appropriate for the current CPU. It could be say LAPACK, BLAS, openmp,
depending of the problem at hand.

Talking about libraries, may be the
https://en.cppreference.com/w/cpp/experimental/simd will do the job?

Best,
Serguei.

   In some cases, code can be rewritten

so that the compiler can vectorize it better, using the level of
vectorized instructions that have been enabled.

Unconditionally using GCC-specific or architecture-specific options in
packages would certainly not be portable. Even on Windows, R is now used
also with clang and on aarch64, so one should not assume a concrete
compiler and architecture.

Please note also that GCC on Windows has a bug due to which AVX2
instructions cannot be used reliably - the compiler doesn't always
properly align local variables on the stack when emitting these. See
[1,2] for more information.

Best
Tomas

[1] https://stat.ethz.ch/pipermail/r-sig-windows/2024q1/000113.html
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412



kind regards,
Jesse

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel




__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Bioc-devel] about deprecate package DNABarcode

[Sun Wenjie]

Hi Pedro,

Tilo has responded and is currently updating the DNABarcode package.

Best,
Wenjie


From: Pedro Sánchez Sánchez 
Sent: Wednesday, March 20, 2024 1:58 PM
To: Kern, Lori
Cc: Sun Wenjie; bioc-devel@r-project.org
Subject: Re: [Bioc-devel] about deprecate package DNABarcode

WARNING: This email originated from outside of Institut Curie. Please validate 
the sender's email address before clicking on links or attachments as they may 
not be safe.

Hi all,

I also tried to contact the author on the available email contacts (waiting for 
response from tilo.buschmann...@gmail.com; 
depreciated email 
tilo.buschm...@izi.fraunhofer.de) I 
found on the internet. If I get a response, I'll let you know.

It will be a pity if DNABarcodes is depreciated for lacking of maintenance.

Just in case, it is helpful for someone, I just want to mention there is an 
apparently similar repository on GitHub with the same goal as DNABarcodes, by 
other developers and written in Python/Jupyter Notebooks: 
https://github.com/feldman4/dna-barcodes.

Best,

Pedro

El mar, 12 mar 2024 a las 13:18, Kern, Lori via Bioc-devel 
(mailto:bioc-devel@r-project.org>>) escribió:
The package is deprecated in devel.  It will remain listed in the next Bioc 
3.19 release, but deprecated, and removed from anticipated 3.20.  The package 
will have up until the fall release (generally late Oct / early Nov) to 
reinstate before it will be removed.

Cheers,


Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263


From: Bioc-devel 
mailto:bioc-devel-boun...@r-project.org>> on 
behalf of Sun Wenjie mailto:wenjie@curie.fr>>
Sent: Tuesday, March 12, 2024 8:07 AM
To: bioc-devel@r-project.org 
mailto:bioc-devel@r-project.org>>
Subject: [Bioc-devel] about deprecate package DNABarcode

Hi Bioconductor core team,

I saw the package of DNABarcode is listed as to be deprecated.

I am trying to reach the author. Do you know when will be the deadline?

Best,
Wenjie

--
Wenjie SUN
Postdoctoral researcher
UMR168, Curie Institute
E-mail: wenjie@curie.fr
Address: batiment Curie, RDC.13A
11 rue Pierre et Marie Curie, 75005 Paris
https://secure-web.cisco.com/1-TBJw4o6EGDXK_14o3XCIp7DQhMLTaxvVR4NCL9q9sGYrycpujC_OYkx7ortrnjf5crYpYXyRsFEcpldkHYYLTfw-V9oZUKv8flhupsI5LV_JzcWwDb1FnvBDTYwcj0cXxmA3GNJQVF3Mnc1QnR55FA-_sO9AKIMHbDHD2U1y9HvFChc3nLydpoM4IZQhFrkC01Z7BtNAdt0gmfJzyF8AM4qlM40FR8g5KAPURr03QM4ukAWrszSyzsi9jEVmrUnL0Tjcsf1thPL1wOBupAU-UZtzsKkLIae_8gglI6k6W8kBMlUjp2SvfS4iEIah835/https%3A%2F%2Fcurie.fr%2Fequipe%2Fperie

___
Bioc-devel@r-project.org mailing list
https://secure-web.cisco.com/1l6kli-5ER6TfFRoWON7eg605W9-aPQ1auinsEup1uN5OFQqRFc_QRRlrkmssHncngEABAwXz4PYezr9ontykNdw3OQlPlaKaQJXx9J_ey5JecbnynELh3K-b_GgnpbL94Du22Rl80OCRrkiU3mGuPhIqnUgFxA9Bo9QsZmZHs3u4yuLvZK9QVQxgvloJXJO8Z0H8rBvABpOzApweEY8hv8ovHZBVkWCCPX1r-yGlLt9G_Y1GJvdqruefoEmWpUBxngKD3odX73chnPmSKwS0X31BlZezoSG7BeE5KK_SR8tKGHjf8oxPHoNBTC1WTP6x/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel



This email message may contain legally privileged and/or confidential 
information.  If you are not the intended recipient(s), or the employee or 
agent responsible for the delivery of this message to the intended 
recipient(s), you are hereby notified that any disclosure, copying, 
distribution, or use of this email message is prohibited.  If you have received 
this message in error, please notify the sender immediately by e-mail and 
delete this email message from your computer. Thank you.
[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [R-pkg-devel] using portable simd instructions

[Vladimir Dergachev]

I like assembler, and I do use SIMD intrinsincs in some of my code (not 
R), but sparingly.


The issue is more than portability between platforms, but also portability 
between processors - if you write your optimized code using AVX, it might 
not take advantage of newer AVX512 cpus.


In many cases your compiler will do the right thing and optimize your 
code.


I suggest:

   * write your code in plain C, test it with some long computation and 
use "perf top" on Linux to observe the code hotspots and which assembler 
instructions are being used.


   * if you see instructions like "addps" these are vectorized. If you see 
instructions like "addss" these are *not* vectorized.


   * if you see a few instructions as hotspots with arguments in 
parenthesis "vmovaps %xmm1,(%r8)" then you are likely limited by memory 
access.


   * If you are not limited by memory access and the compiler produces a 
lot of "addss" or similar that are hotspots, then you need to look at your 
code and make it more parallelizable.


   * How to make your C code more parallelizable:

   You want to make easy to interpret loops like

 for(i=start;i   You can help the compiler by using "restrict" keyword to indicate that 
arrays do not overlap, or (as a sledgehammer) "#pragma ivdep". But before 
using keywords check with "perf top" which code is actually a hotspot, as 
the compiler can generate good code without restrict keywords, by using 
multiple code paths.


   * You can create small temporary arrays to make your algorithm look 
more like loops above. The small arrays should be at least 16 wide, 
because AVX512 has instructions that operate on 16 floats at a time.


* To allow use of small arrays you can unroll your loops. Note that 
compilers do unrolling themselves, so doing it manually is only helpful if 
this makes the inner body of the loop more parallelizable.


* You can debug why the compiler does not parallelize your code by 
turning on diagnostics. For gcc the flag is "-fopt-info-vec-missed=vec_info.txt"


* In very rare cases you use intrinsics. For me this is typically a 
situation when I need to find a value and the index of a maximum or 
minimum in an array - compilers do not optimize this well, at least for 
many different ways of coding this in C that I have tried many years ago.


* If after all your work you got a factor of 2 speedup you are doing 
fine. If you want larger speedup change your algorithm.


best

Vladimir Dergachev

On Wed, 27 Mar 2024, Dirk Eddelbuettel wrote:



On 27 March 2024 at 08:48, jesse koops wrote:
| Thank you, I was not aware of the easy way to search CRAN. I looked at
| rcppsimdjson of course, but couldn't figure it out since it is done in
| the simdjson library if interpret it correclty, not within the R
| ecosystem and I didn't know how that would change things. Writing R
| extensions assumes a lot of  prior knowledge so I will have to work my
| way up to there first.

I think I have (at least) one other package doing something like this _in the
library layer too_ as suggested by Tomas, namely crc32c as used by digest.
You could study how crc32c [0] does this for x86_64 and arm64 to get hardware
optimization. (This may be more specific cpu hardware optimization but at
least the library and cmake files are small.)

I decided as a teenager that assembler wasn't for me and haven't looked back,
but I happily take advantage of it when bundled well. So strong second for
the recommendation by Tomas to rely on this being done in an external and
tested library.

(Another interesting one there is highway [1]. Just packaging that would
likely be an excellent contribution.)

Dirk

[0] repo: https://github.com/google/crc32c
[1] repo: https://github.com/google/highway
   docs: https://google.github.io/highway/en/master/


|
| Op di 26 mrt 2024 om 15:41 schreef Dirk Eddelbuettel :
| >
| >
| > On 26 March 2024 at 10:53, jesse koops wrote:
| > | How can I make this portable and CRAN-acceptable?
| >
| > But writing (or borrowing ?) some hardware detection via either configure /
| > autoconf or cmake. This is no different than other tasks decided at 
install-time.
| >
| > Start with 'Writing R Extensions', as always, and work your way up from
| > there. And if memory serves there are already a few other packages with SIMD
| > at CRAN so you can also try to take advantage of the search for a 'token'
| > (here: 'SIMD') at the (unofficial) CRAN mirror at GitHub:
| >
| >https://github.com/search?q=org%3Acran%20SIMD=code
| >
| > Hth, Dirk
| >
| > --
| > dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

--
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel



__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] using portable simd instructions

[jesse koops]

I tried that but I found the interface awkward and there was really no
performance bonus. It was in the early phase of experimentation and I
didn't save it, so it could very well be that I got the compiler
settings wrong and simd was not used. But if that was the case, there
would still be the problem of using the correct compiler settings
cross platform.

Op wo 27 mrt 2024 om 14:44 schreef Serguei Sokol :
>
> Le 26/03/2024 à 15:51, Tomas Kalibera a écrit :
> >
> > On 3/26/24 10:53, jesse koops wrote:
> >> Hello R-package-devel,
> >>
> >> I recently got inspired by the rcppsimdjson package to try out simd
> >> registers. It works fantastic on my computer but I struggle to find
> >> information on how to make it portable. It doesn't help in this case
> >> that R and Rcpp make including Cpp code so easy that I have never had
> >> to learn about cmake and compiler flags. I would appreciate any help,
> >> including of the type: "go read instructions at ...".
> >>
> >> I use RcppArmadillo and Rcpp. I currenlty include the following header:
> >>
> >> #include 
> >>
> >> The functions in immintrin that I use are:
> >>
> >> _mm256_loadu_pd
> >> _mm256_set1_pd
> >> _mm256_mul_pd
> >> _mm256_fmadd_pd
> >> _mm256_storeu_pd
> >>
> >> and I define up to four __m256d registers. From information found
> >> online (not sure where anymore) I constructed the following makevars
> >> file:
> >>
> >> CXX_STD = CXX14
> >>
> >> PKG_CPPFLAGS = -I../inst/include -mfma -msse4.2 -mavx
> >>
> >> PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
> >> PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
> >>
> >> (I also use openmp, that has always worked fine, I just included all
> >> lines for completeness) Rcheck gives me two notes:
> >>
> >> ─  using R version 4.3.2 (2023-10-31 ucrt)
> >> ─  using platform: x86_64-w64-mingw32 (64-bit)
> >> ─  R was compiled by
> >> gcc.exe (GCC) 12.3.0
> >> GNU Fortran (GCC) 12.3.0
> >>
> >> ❯ checking compilation flags used ... NOTE
> >>Compilation used the following non-portable flag(s):
> >>  '-mavx' '-mfma' '-msse4.2'
> >>
> >> ❯ checking C++ specification ... NOTE
> >>  Specified C++14: please drop specification unless essential
> >>
> >> But as far as I understand, the flags are necessary, at least in GCC.
> >> How can I make this portable and CRAN-acceptable?
> >
> > I think it the best way for portability is to use a higher-level library
> > that already has done the low-level business of maintaining multiple
> > versions of the code (with multiple instruction sets) and choosing one
> > appropriate for the current CPU. It could be say LAPACK, BLAS, openmp,
> > depending of the problem at hand.
> Talking about libraries, may be the
> https://en.cppreference.com/w/cpp/experimental/simd will do the job?
>
> Best,
> Serguei.
>
>   In some cases, code can be rewritten
> > so that the compiler can vectorize it better, using the level of
> > vectorized instructions that have been enabled.
> >
> > Unconditionally using GCC-specific or architecture-specific options in
> > packages would certainly not be portable. Even on Windows, R is now used
> > also with clang and on aarch64, so one should not assume a concrete
> > compiler and architecture.
> >
> > Please note also that GCC on Windows has a bug due to which AVX2
> > instructions cannot be used reliably - the compiler doesn't always
> > properly align local variables on the stack when emitting these. See
> > [1,2] for more information.
> >
> > Best
> > Tomas
> >
> > [1] https://stat.ethz.ch/pipermail/r-sig-windows/2024q1/000113.html
> > [2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
> >
> >>
> >> kind regards,
> >> Jesse
> >>
> >> __
> >> R-package-devel@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
> > __
> > R-package-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] using portable simd instructions

[Serguei Sokol]

Le 26/03/2024 à 15:51, Tomas Kalibera a écrit :


On 3/26/24 10:53, jesse koops wrote:

Hello R-package-devel,

I recently got inspired by the rcppsimdjson package to try out simd
registers. It works fantastic on my computer but I struggle to find
information on how to make it portable. It doesn't help in this case
that R and Rcpp make including Cpp code so easy that I have never had
to learn about cmake and compiler flags. I would appreciate any help,
including of the type: "go read instructions at ...".

I use RcppArmadillo and Rcpp. I currenlty include the following header:

#include 

The functions in immintrin that I use are:

_mm256_loadu_pd
_mm256_set1_pd
_mm256_mul_pd
_mm256_fmadd_pd
_mm256_storeu_pd

and I define up to four __m256d registers. From information found
online (not sure where anymore) I constructed the following makevars
file:

CXX_STD = CXX14

PKG_CPPFLAGS = -I../inst/include -mfma -msse4.2 -mavx

PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)

(I also use openmp, that has always worked fine, I just included all
lines for completeness) Rcheck gives me two notes:

─  using R version 4.3.2 (2023-10-31 ucrt)
─  using platform: x86_64-w64-mingw32 (64-bit)
─  R was compiled by
    gcc.exe (GCC) 12.3.0
    GNU Fortran (GCC) 12.3.0

❯ checking compilation flags used ... NOTE
   Compilation used the following non-portable flag(s):
 '-mavx' '-mfma' '-msse4.2'

❯ checking C++ specification ... NOTE
 Specified C++14: please drop specification unless essential

But as far as I understand, the flags are necessary, at least in GCC.
How can I make this portable and CRAN-acceptable?


I think it the best way for portability is to use a higher-level library 
that already has done the low-level business of maintaining multiple 
versions of the code (with multiple instruction sets) and choosing one 
appropriate for the current CPU. It could be say LAPACK, BLAS, openmp, 
depending of the problem at hand.
Talking about libraries, may be the 
https://en.cppreference.com/w/cpp/experimental/simd will do the job?


Best,
Serguei.

 In some cases, code can be rewritten
so that the compiler can vectorize it better, using the level of 
vectorized instructions that have been enabled.


Unconditionally using GCC-specific or architecture-specific options in 
packages would certainly not be portable. Even on Windows, R is now used 
also with clang and on aarch64, so one should not assume a concrete 
compiler and architecture.


Please note also that GCC on Windows has a bug due to which AVX2 
instructions cannot be used reliably - the compiler doesn't always 
properly align local variables on the stack when emitting these. See 
[1,2] for more information.


Best
Tomas

[1] https://stat.ethz.ch/pipermail/r-sig-windows/2024q1/000113.html
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412



kind regards,
Jesse

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Rd] paths capability FALSE on devel?

[Alexandre Courtiol]

On Wed, 27 Mar 2024 at 12:19, Ivan Krylov  wrote:

> В Wed, 27 Mar 2024 11:28:17 +0100
> Alexandre Courtiol  пишет:
>
> > after installing R-devel the output of
> > grDevices::dev.capabilities()$paths is FALSE, while it is TRUE for R
> > 4.3.3
>
> Your system must be missing Cairo development headers, making x11()
> fall back to type = 'Xlib':
>
> $ R-devel -q -s -e 'x11(); grDevices::dev.capabilities()$paths'
>  [1] TRUE
> $ R-devel -q -s -e \
>  'x11(type="Xlib"); grDevices::dev.capabilities()$paths'
>  [1] FALSE
>
> If that's not the case and capabilities()['cairo'] is TRUE in your
> build of R-devel, please show us the sessionInfo() from your build of
> R-devel.
>
> --
> Best regards,
> Ivan
>

Thanks everyone for your feedback.
Here is the requested information:

> x11()
> grDevices::dev.capabilities()$paths
[1] FALSE
> capabilities()['cairo']
cairo
TRUE
> sessionInfo()
R version 4.3.3 Patched (2024-02-29 r86166)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS:   /home/courtiol/R-devel/lib/R/lib/libRblas.so
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
[1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8   LC_NAME=C
[9] LC_ADDRESS=C   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_4.3.3

However, since it works on your side, I think there is nothing to worry
about, I must have simply failed to install R-devel properly.

-- 
Alexandre Courtiol, www.datazoogang.de

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] using portable simd instructions

[jesse koops]

Thanks, the source of digest seems especially helpful. Ironically, I
actually used only 10 or so lines with simd code in the package in a
single function that is a bottleneck. It gives a very noticable
performance boost, even with unaligned vectors, and something about
directly using processor registers seems elegant to me. The article by
Langdale and Lemire that you pointed to on your blog was also
inspirational.

Unfortunately getting those 10 lines to reliably work on other
machines looks like a long term project and the advice from Thomas
that GCC on windows might decide to faultily optimize other code is a
bit scary.  I'll take the advice to heart and will not try to include
simd code in any of my cran packages in the near future. I learned C
and Cpp via R, like many people probably,  and makefiles and
portability have never come up until now so there's quite some
learning ahead. At least I now have some good places to start. Thank
you all for the help!

Op wo 27 mrt 2024 om 12:27 schreef Dirk Eddelbuettel :
>
>
> On 27 March 2024 at 08:48, jesse koops wrote:
> | Thank you, I was not aware of the easy way to search CRAN. I looked at
> | rcppsimdjson of course, but couldn't figure it out since it is done in
> | the simdjson library if interpret it correclty, not within the R
> | ecosystem and I didn't know how that would change things. Writing R
> | extensions assumes a lot of  prior knowledge so I will have to work my
> | way up to there first.
>
> I think I have (at least) one other package doing something like this _in the
> library layer too_ as suggested by Tomas, namely crc32c as used by digest.
> You could study how crc32c [0] does this for x86_64 and arm64 to get hardware
> optimization. (This may be more specific cpu hardware optimization but at
> least the library and cmake files are small.)
>
> I decided as a teenager that assembler wasn't for me and haven't looked back,
> but I happily take advantage of it when bundled well. So strong second for
> the recommendation by Tomas to rely on this being done in an external and
> tested library.
>
> (Another interesting one there is highway [1]. Just packaging that would
> likely be an excellent contribution.)
>
> Dirk
>
> [0] repo: https://github.com/google/crc32c
> [1] repo: https://github.com/google/highway
> docs: https://google.github.io/highway/en/master/
>
>
> |
> | Op di 26 mrt 2024 om 15:41 schreef Dirk Eddelbuettel :
> | >
> | >
> | > On 26 March 2024 at 10:53, jesse koops wrote:
> | > | How can I make this portable and CRAN-acceptable?
> | >
> | > But writing (or borrowing ?) some hardware detection via either configure 
> /
> | > autoconf or cmake. This is no different than other tasks decided at 
> install-time.
> | >
> | > Start with 'Writing R Extensions', as always, and work your way up from
> | > there. And if memory serves there are already a few other packages with 
> SIMD
> | > at CRAN so you can also try to take advantage of the search for a 'token'
> | > (here: 'SIMD') at the (unofficial) CRAN mirror at GitHub:
> | >
> | >https://github.com/search?q=org%3Acran%20SIMD=code
> | >
> | > Hth, Dirk
> | >
> | > --
> | > dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
>
> --
> dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[Bioc-devel] fixing deltaCaptureC

[Michael Shapiro]

My package deltaCaptureC is suddenly having build problems, specifically a 
problem building the vignette.  It fails when attempting to display a ggplot 
object in the vignette, and this seems to be due to changes in ggplot.  I've 
updated the .rda files for the plots and the vignette now builds properly 
locally.  I bumped the version and pushed to RELEASE_3_18.  I did this last 
week, but yesterday I got an email that it is still failing to build.

Please advise.

Many thanks,
Michael


The Francis Crick Institute Limited is a registered charity in England and 
Wales no. 1140062 and a company registered in England and Wales no. 06885462, 
with its registered office at 1 Midland Road London NW1 1AT

[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [R-pkg-devel] using portable simd instructions

[Dirk Eddelbuettel]

On 27 March 2024 at 08:48, jesse koops wrote:
| Thank you, I was not aware of the easy way to search CRAN. I looked at
| rcppsimdjson of course, but couldn't figure it out since it is done in
| the simdjson library if interpret it correclty, not within the R
| ecosystem and I didn't know how that would change things. Writing R
| extensions assumes a lot of  prior knowledge so I will have to work my
| way up to there first.

I think I have (at least) one other package doing something like this _in the
library layer too_ as suggested by Tomas, namely crc32c as used by digest.
You could study how crc32c [0] does this for x86_64 and arm64 to get hardware
optimization. (This may be more specific cpu hardware optimization but at
least the library and cmake files are small.)

I decided as a teenager that assembler wasn't for me and haven't looked back,
but I happily take advantage of it when bundled well. So strong second for
the recommendation by Tomas to rely on this being done in an external and
tested library.

(Another interesting one there is highway [1]. Just packaging that would
likely be an excellent contribution.)

Dirk

[0] repo: https://github.com/google/crc32c
[1] repo: https://github.com/google/highway
docs: https://google.github.io/highway/en/master/


| 
| Op di 26 mrt 2024 om 15:41 schreef Dirk Eddelbuettel :
| >
| >
| > On 26 March 2024 at 10:53, jesse koops wrote:
| > | How can I make this portable and CRAN-acceptable?
| >
| > But writing (or borrowing ?) some hardware detection via either configure /
| > autoconf or cmake. This is no different than other tasks decided at 
install-time.
| >
| > Start with 'Writing R Extensions', as always, and work your way up from
| > there. And if memory serves there are already a few other packages with SIMD
| > at CRAN so you can also try to take advantage of the search for a 'token'
| > (here: 'SIMD') at the (unofficial) CRAN mirror at GitHub:
| >
| >https://github.com/search?q=org%3Acran%20SIMD=code
| >
| > Hth, Dirk
| >
| > --
| > dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

-- 
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [Bioc-devel] Commit access for additional developer on missMethyl package

[Kern, Lori via Bioc-devel]

I'll be in touch with Calandra privately to set up a BiocCredentials account. 
Once that is established we can proceed with this request


Lori Shepherd - Kern

Bioconductor Core Team

Roswell Park Comprehensive Cancer Center

Department of Biostatistics & Bioinformatics

Elm & Carlton Streets

Buffalo, New York 14263


From: Bioc-devel  on behalf of Jovana 
Maksimovic via Bioc-devel 
Sent: Wednesday, March 27, 2024 1:16 AM
To: bioc-devel@r-project.org 
Subject: [Bioc-devel] Commit access for additional developer on missMethyl 
package

Hi Bioconductor Team,
I would like to organise commit access for a new developer on the missMethyl 
package? Calandra Grima 
(calandra.gr...@petermac.org) will be 
joining the maintenance team for the package and Andrew Lonsdale can be removed.
Please let me know if there is any additional information you require for this 
process.
Kind regards,
Jovana Maksimovic


[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://secure-web.cisco.com/1BjSwJZw6zbJt2bjTn9jpcek2WOtwE6OUO4qV7S5VyoPVHiAwSUIBKVk0_m7th21bh6DSnRkv0Zb5_5HXrZabMO0ktVHB7V2e_zSlRCxdb0jkEQiuDmCSjMxKTuuGjRRjs82tHVI4mtvCOn2VHeEwXv7rXZydDrVqZlKn4PxMZe0TZ0P99_wvSGpFoqRxpspeoAF5fBqe8FHZ5tVDrfZrEXsiESs0RMKDCKmAho7zAJBcYCRLTmCq5kXYNfMj2VshZ9PxIvqxthBoSk993s8UQRHY-dN3aMrH4PT6LQLPOZAXszxA0QQJNaQIz3dCtpE3/https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fbioc-devel



This email message may contain legally privileged and/or confidential 
information.  If you are not the intended recipient(s), or the employee or 
agent responsible for the delivery of this message to the intended 
recipient(s), you are hereby notified that any disclosure, copying, 
distribution, or use of this email message is prohibited.  If you have received 
this message in error, please notify the sender immediately by e-mail and 
delete this email message from your computer. Thank you.
[[alternative HTML version deleted]]

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel

Re: [Rd] paths capability FALSE on devel?

[Ivan Krylov via R-devel]

В Wed, 27 Mar 2024 11:28:17 +0100
Alexandre Courtiol  пишет:

> after installing R-devel the output of
> grDevices::dev.capabilities()$paths is FALSE, while it is TRUE for R
> 4.3.3

Your system must be missing Cairo development headers, making x11()
fall back to type = 'Xlib':

$ R-devel -q -s -e 'x11(); grDevices::dev.capabilities()$paths'
 [1] TRUE
$ R-devel -q -s -e \
 'x11(type="Xlib"); grDevices::dev.capabilities()$paths'
 [1] FALSE

If that's not the case and capabilities()['cairo'] is TRUE in your
build of R-devel, please show us the sessionInfo() from your build of
R-devel.

-- 
Best regards,
Ivan

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] paths capability FALSE on devel?

[Dirk Eddelbuettel]

On 27 March 2024 at 11:03, Prof Brian Ripley via R-devel wrote:
| On 27/03/2024 10:28, Alexandre Courtiol wrote:
| > Hi all,
| > 
| > I don't know if it is a local issue on my hands or not, but after
| > installing R-devel the output of grDevices::dev.capabilities()$paths is
| > FALSE, while it is TRUE for R 4.3.3.
| > Relatedly, I have issues with plotting paths on devel.
| > 
| > At this stage, I simply would like to know if others running R devel and R
| > 4.3.3 can replicate this behaviour and if there are obvious reasons why the
| > observed change would be expected.
| 
| The help says
| 
|   Query the capabilities of the current graphics device.
| 
| You haven't told us what that was.  See the posting guide for the "at a 
| minimum" information you also did not provide 

Yes, with that I see

> x11()
> grDevices::dev.capabilities()$paths
[1] TRUE
>
> getRversion()
[1] ‘4.5.0’
>
> R.version
   _ 
platform   x86_64-pc-linux-gnu   
arch   x86_64
os linux-gnu 
system x86_64, linux-gnu 
status Under development (unstable)  
major  4 
minor  5.0   
year   2024  
month  03
day27
svn rev86214 
language   R 
version.string R Under development (unstable) (2024-03-27 r86214)
nickname   Unsuffered Consequences   
> 

Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] paths capability FALSE on devel?

[Prof Brian Ripley via R-devel]

On 27/03/2024 10:28, Alexandre Courtiol wrote:

Hi all,

I don't know if it is a local issue on my hands or not, but after
installing R-devel the output of grDevices::dev.capabilities()$paths is
FALSE, while it is TRUE for R 4.3.3.
Relatedly, I have issues with plotting paths on devel.

At this stage, I simply would like to know if others running R devel and R
4.3.3 can replicate this behaviour and if there are obvious reasons why the
observed change would be expected.


The help says

 Query the capabilities of the current graphics device.

You haven't told us what that was.  See the posting guide for the "at a 
minimum" information you also did not provide 



--
Brian D. Ripley,  rip...@stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] paths capability FALSE on devel?

[Alexandre Courtiol]

Hi all,

I don't know if it is a local issue on my hands or not, but after
installing R-devel the output of grDevices::dev.capabilities()$paths is
FALSE, while it is TRUE for R 4.3.3.
Relatedly, I have issues with plotting paths on devel.

At this stage, I simply would like to know if others running R devel and R
4.3.3 can replicate this behaviour and if there are obvious reasons why the
observed change would be expected.

Man thanks,

Alex
-- 
Alexandre Courtiol, www.datazoogang.de

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [R-pkg-devel] Fwd: using portable simd instructions

[Tomas Kalibera]

On 3/27/24 08:39, jesse koops wrote:

Of course you are correct about the portability. But since at ;least
one other CRAN package by a renowned author does it succesfully, I
figured I'd experiment first on my machine and learn about portability
later. Thank you for the links and the warning about the bug. I was
aware of that, however I am careful to only use the "loadu" and
"storeu" variants, so I thought this would not bite me. Do you know if
my assumption is in error?


My advice is please do not publish any packages doing this low level 
stuff unless you fully understand the details yourself. If you don't, 
please work at a higher level abstraction and use existing code for the 
low-level things, to avoid adding to the maintenance costs. These things 
can take very long to debug.


The GCC bug on Windows I've ran into only affects instructions that 
require aligned operands (on the stack), aligned at 32-byte boundary.


Tomas



Op di 26 mrt 2024 om 15:51 schreef Tomas Kalibera :


On 3/26/24 10:53, jesse koops wrote:

Hello R-package-devel,

I recently got inspired by the rcppsimdjson package to try out simd
registers. It works fantastic on my computer but I struggle to find
information on how to make it portable. It doesn't help in this case
that R and Rcpp make including Cpp code so easy that I have never had
to learn about cmake and compiler flags. I would appreciate any help,
including of the type: "go read instructions at ...".

I use RcppArmadillo and Rcpp. I currenlty include the following header:

#include 

The functions in immintrin that I use are:

_mm256_loadu_pd
_mm256_set1_pd
_mm256_mul_pd
_mm256_fmadd_pd
_mm256_storeu_pd

and I define up to four __m256d registers. From information found
online (not sure where anymore) I constructed the following makevars
file:

CXX_STD = CXX14

PKG_CPPFLAGS = -I../inst/include -mfma -msse4.2 -mavx

PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)

(I also use openmp, that has always worked fine, I just included all
lines for completeness) Rcheck gives me two notes:

─  using R version 4.3.2 (2023-10-31 ucrt)
─  using platform: x86_64-w64-mingw32 (64-bit)
─  R was compiled by
 gcc.exe (GCC) 12.3.0
 GNU Fortran (GCC) 12.3.0

❯ checking compilation flags used ... NOTE
Compilation used the following non-portable flag(s):
  '-mavx' '-mfma' '-msse4.2'

❯ checking C++ specification ... NOTE
  Specified C++14: please drop specification unless essential

But as far as I understand, the flags are necessary, at least in GCC.
How can I make this portable and CRAN-acceptable?

I think it the best way for portability is to use a higher-level library
that already has done the low-level business of maintaining multiple
versions of the code (with multiple instruction sets) and choosing one
appropriate for the current CPU. It could be say LAPACK, BLAS, openmp,
depending of the problem at hand. In some cases, code can be rewritten
so that the compiler can vectorize it better, using the level of
vectorized instructions that have been enabled.

Unconditionally using GCC-specific or architecture-specific options in
packages would certainly not be portable. Even on Windows, R is now used
also with clang and on aarch64, so one should not assume a concrete
compiler and architecture.

Please note also that GCC on Windows has a bug due to which AVX2
instructions cannot be used reliably - the compiler doesn't always
properly align local variables on the stack when emitting these. See
[1,2] for more information.

Best
Tomas

[1] https://stat.ethz.ch/pipermail/r-sig-windows/2024q1/000113.html
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412


kind regards,
Jesse

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] using portable simd instructions

[jesse koops]

Thank you, I was not aware of the easy way to search CRAN. I looked at
rcppsimdjson of course, but couldn't figure it out since it is done in
the simdjson library if interpret it correclty, not within the R
ecosystem and I didn't know how that would change things. Writing R
extensions assumes a lot of  prior knowledge so I will have to work my
way up to there first.

Op di 26 mrt 2024 om 15:41 schreef Dirk Eddelbuettel :
>
>
> On 26 March 2024 at 10:53, jesse koops wrote:
> | How can I make this portable and CRAN-acceptable?
>
> But writing (or borrowing ?) some hardware detection via either configure /
> autoconf or cmake. This is no different than other tasks decided at 
> install-time.
>
> Start with 'Writing R Extensions', as always, and work your way up from
> there. And if memory serves there are already a few other packages with SIMD
> at CRAN so you can also try to take advantage of the search for a 'token'
> (here: 'SIMD') at the (unofficial) CRAN mirror at GitHub:
>
>https://github.com/search?q=org%3Acran%20SIMD=code
>
> Hth, Dirk
>
> --
> dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[R-pkg-devel] Fwd: using portable simd instructions

[jesse koops]

Of course you are correct about the portability. But since at ;least
one other CRAN package by a renowned author does it succesfully, I
figured I'd experiment first on my machine and learn about portability
later. Thank you for the links and the warning about the bug. I was
aware of that, however I am careful to only use the "loadu" and
"storeu" variants, so I thought this would not bite me. Do you know if
my assumption is in error?

Op di 26 mrt 2024 om 15:51 schreef Tomas Kalibera :
>
>
> On 3/26/24 10:53, jesse koops wrote:
> > Hello R-package-devel,
> >
> > I recently got inspired by the rcppsimdjson package to try out simd
> > registers. It works fantastic on my computer but I struggle to find
> > information on how to make it portable. It doesn't help in this case
> > that R and Rcpp make including Cpp code so easy that I have never had
> > to learn about cmake and compiler flags. I would appreciate any help,
> > including of the type: "go read instructions at ...".
> >
> > I use RcppArmadillo and Rcpp. I currenlty include the following header:
> >
> > #include 
> >
> > The functions in immintrin that I use are:
> >
> > _mm256_loadu_pd
> > _mm256_set1_pd
> > _mm256_mul_pd
> > _mm256_fmadd_pd
> > _mm256_storeu_pd
> >
> > and I define up to four __m256d registers. From information found
> > online (not sure where anymore) I constructed the following makevars
> > file:
> >
> > CXX_STD = CXX14
> >
> > PKG_CPPFLAGS = -I../inst/include -mfma -msse4.2 -mavx
> >
> > PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
> > PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
> >
> > (I also use openmp, that has always worked fine, I just included all
> > lines for completeness) Rcheck gives me two notes:
> >
> > ─  using R version 4.3.2 (2023-10-31 ucrt)
> > ─  using platform: x86_64-w64-mingw32 (64-bit)
> > ─  R was compiled by
> > gcc.exe (GCC) 12.3.0
> > GNU Fortran (GCC) 12.3.0
> >
> > ❯ checking compilation flags used ... NOTE
> >Compilation used the following non-portable flag(s):
> >  '-mavx' '-mfma' '-msse4.2'
> >
> > ❯ checking C++ specification ... NOTE
> >  Specified C++14: please drop specification unless essential
> >
> > But as far as I understand, the flags are necessary, at least in GCC.
> > How can I make this portable and CRAN-acceptable?
>
> I think it the best way for portability is to use a higher-level library
> that already has done the low-level business of maintaining multiple
> versions of the code (with multiple instruction sets) and choosing one
> appropriate for the current CPU. It could be say LAPACK, BLAS, openmp,
> depending of the problem at hand. In some cases, code can be rewritten
> so that the compiler can vectorize it better, using the level of
> vectorized instructions that have been enabled.
>
> Unconditionally using GCC-specific or architecture-specific options in
> packages would certainly not be portable. Even on Windows, R is now used
> also with clang and on aarch64, so one should not assume a concrete
> compiler and architecture.
>
> Please note also that GCC on Windows has a bug due to which AVX2
> instructions cannot be used reliably - the compiler doesn't always
> properly align local variables on the stack when emitting these. See
> [1,2] for more information.
>
> Best
> Tomas
>
> [1] https://stat.ethz.ch/pipermail/r-sig-windows/2024q1/000113.html
> [2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
>
> >
> > kind regards,
> > Jesse
> >
> > __
> > R-package-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] using portable simd instructions

[jesse koops]

Thank you very much, that looks promising. Though if I look at your
congigure.ac script, also extremely daunting and far above my current
level of understanding. I guess I'll start with the autoconf manual
then.

Op di 26 mrt 2024 om 16:04 schreef Vincent Dorie :
>
> Hi Jesse,
>
> What I've done is to use a mix of compile-time detection of compiler SIMD 
> support and run-time detection of SIMD hardware support. At package load, 
> SIMD-specific versions of functions are installed in a symbol table. It's not 
> perfect and it can be hard to support evolving platforms, especially now that 
> ARM is more prevalent. However, it does allow for distribution on CRAN as it 
> uses only autoconf, POSIX make, and no specific compiler.
>
> At compile time:
> 1. Use a configure script to detect the platform and any SIMD instructions 
> supported by the compiler. This is also the time to identify the compiler 
> flags necessary to enable instruction sets. Unlike what the existing autoconf 
> macros do, you can ignore whether or not the host system supports the 
> instruction sets (with the exception when compiling with Solaris Studio - it 
> won't let you load a binary with instructions not supported by the host, even 
> if they cannot be executed).
> 2. Use makefiles to conditionally compile different versions of the functions 
> you want, one for each level of instruction set supported by the compiler, 
> using the flags detected above. They all should be in different files with 
> different symbols. For example: partition_sse2.c defines partition_sse2(), 
> partition_avx.c defines partition_avx(), etc., while partition.c defines 
> partition_c() - a fall-back compiled without any SIMD instructions. Note that 
> echoing compilations with SIMD flags will trigger a check warning, as those 
> units are not inherently portable. That is addressed below.
>
> At run time:
> 1. On package load, detect what instruction sets are supported by the host. 
> On x86 machines, this usually involves a call to cpuid.
> 2. For the maximum level of instruction set supported by the host, install 
> the relevant symbol for each function into a symbol table. Using the example 
> above, a header defines an external function pointer partition(), which gets 
> set to one of the SIMD-specific implementations.
>
> In setting that up, I found Agner Fog's notes on CPU dispatching to be 
> extremely helpful. They can be found here: https://www.agner.org/optimize. I 
> use this strategy in the dbarts package, the code for which is here: 
> https://github.com/vdorie/dbarts.
>
> Best,
> Vince
>
> On Tue, Mar 26, 2024 at 10:45 AM Dirk Eddelbuettel  wrote:
>>
>>
>> On 26 March 2024 at 10:53, jesse koops wrote:
>> | How can I make this portable and CRAN-acceptable?
>>
>> But writing (or borrowing ?) some hardware detection via either configure /
>> autoconf or cmake. This is no different than other tasks decided at 
>> install-time.
>>
>> Start with 'Writing R Extensions', as always, and work your way up from
>> there. And if memory serves there are already a few other packages with SIMD
>> at CRAN so you can also try to take advantage of the search for a 'token'
>> (here: 'SIMD') at the (unofficial) CRAN mirror at GitHub:
>>
>>https://github.com/search?q=org%3Acran%20SIMD=code
>>
>> Hth, Dirk
>>
>> --
>> dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
>>
>> __
>> R-package-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-package-devel

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel