Re: [Rd] [External] GC: speeding-up the CHARSXP cache maintenance, 2nd try

2021-11-07 Thread Andreas Kersting
enerations that > may be simpler. > > Best, > > luke > > On Wed, 3 Nov 2021, Andreas Kersting wrote: > >> Hi, >> >> In https://stat.ethz.ch/pipermail/r-devel/2021-October/081147.html I >> proposed to speed up the CHARSXP cache maintenance during GC using

Re: [Rd] GC: parallelizing the CHARSXP cache maintenance

2021-11-07 Thread Andreas Kersting
I have now added this to the wishlist in Bugzilla: https://bugs.r-project.org/show_bug.cgi?id=18234 2021-10-07 09:26 GMT+02:00 "Andreas Kersting" : > Hi all, > > As part of RunGenCollect() (in src/main/memory.c), some maintenance on the > CHARSXP cache is done, namely u

[Rd] GC: speeding-up the CHARSXP cache maintenance, 2nd try

2021-11-04 Thread Andreas Kersting
Hi, In https://stat.ethz.ch/pipermail/r-devel/2021-October/081147.html I proposed to speed up the CHARSXP cache maintenance during GC using threading. This was rejected by Luke in https://stat.ethz.ch/pipermail/r-devel/2021-October/081172.html. Here I want to propose an alternative approach

[Rd] GC: improving the marking performance for STRSXPs

2021-10-07 Thread Andreas Kersting
Hi all, in GC (in src/main/memory.c), FORWARD_CHILDREN() (called by PROCESS_NODES()) treats STRSXPs just like VECSXPs, i.e. it calls FORWARD_NODE() for all its children. I claim that this is unnecessarily inefficient since the children of a STRSXP can legitimately only be (atomic) CHARSXPs and

[Rd] GC: parallelizing the CHARSXP cache maintenance

2021-10-07 Thread Andreas Kersting
Hi all, As part of RunGenCollect() (in src/main/memory.c), some maintenance on the CHARSXP cache is done, namely unmarked nodes/CHARSXPs are removed from the hash chains. This requires always touching all CHARSXP in the cache, irrespective of the number of generations which were just garbage

Re: [Rd] [External] memory consumption of nested (un)serialize of sys.frames()

2021-04-08 Thread Andreas Kersting
4-07 17:09 GMT+02:00 "Andreas Kersting" : > Hi Dirk, hi Luke, > > Thanks for checking! > > I could narrow it down further. I have the issue only if I install > --with-keep.source, i.e. > > R CMD INSTALL --with-keep.source dumpTest > > Since this is t

Re: [Rd] [External] memory consumption of nested (un)serialize of sys.frames()

2021-04-07 Thread Andreas Kersting
also from base R. If I install using e.g. devtools::install_github() directly it is also fine for me. Could you please confirm? Thanks! Regards, Andreas 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" : > > On 7 April 2021 at 16:06, Andreas Kersting wrote: > | Hi Luke, > | &

Re: [Rd] [External] memory consumption of nested (un)serialize of sys.frames()

2021-04-07 Thread Andreas Kersting
9 user system elapsed 11.807 0.708 12.597 [1] 10 ^C Timing stopped at: 6.638 0.549 7.214 I had to interrupt in iteration 10 because I was running low on RAM. Regards, Andreas 2021-04-07 15:28 GMT+02:00 luke-tier...@uiowa.edu: > On Wed, 7 Apr 2021, Andreas Kersting wrote: > >

[Rd] memory consumption of nested (un)serialize of sys.frames()

2021-04-07 Thread Andreas Kersting
Hi, please consider the following minimal reproducible example: Create a new R package which just contains the following two (exported) objects: crash_dumps <- new.env() f <- function() { x <- runif(1e5) dump <- lapply(1:2, function(i) unserialize(serialize(sys.frames(), NULL)))

Re: [Rd] custom allocators, Valgrind and uninitialized memory

2021-03-30 Thread Andreas Kersting
the memory content is only > valid (initialised) if you used mmap with previously initialised content. > Again, entirely up to you to decide what the semantics are since you are the > author of the custom allocator. Does that make sense? > > Cheers, > Simon > > > >> On M

Re: [Rd] custom allocators, Valgrind and uninitialized memory

2021-03-29 Thread Andreas Kersting
lagging. You simply have to call VALGRIND_MAKE_MEM_DEFINED() in > your code after allocVector3() to declare that you have initialised the > memory - or am I missing something? > > Cheers, > Simon > > > >> On 30/03/2021, at 9:18 AM, Andreas Kersting wrote: >> >> Hi T

Re: [Rd] custom allocators, Valgrind and uninitialized memory

2021-03-29 Thread Andreas Kersting
y directly from allocVector3(). Regards, Andreas 2021-03-29 10:41 GMT+02:00 "Tomas Kalibera" : > Hi Andreas, > On 3/26/21 8:48 PM, Andreas Kersting wrote: >> Hi Dirk, > > Sure, let me try to explain: > > CRAN ran the tests of my >> package using R which wa

Re: [Rd] custom allocators, Valgrind and uninitialized memory

2021-03-26 Thread Andreas Kersting
Hi Dirk, Sure, let me try to explain: CRAN ran the tests of my package using R which was configured --with-valgrind-instrumentation > 0. Valgrind reported many errors related to the use of supposedly uninitialized memory and the CRAN team asked me to tackle these. These errors are false

Re: [Rd] custom allocators, Valgrind and uninitialized memory

2021-03-26 Thread Andreas Kersting
!? Not sure though if Oracle Developer Studio on Solaris supports __has_include() ... 2021-03-26 08:40 GMT+01:00 "Andreas Kersting" : > Hi, > > In my package bettermc, I use a custom allocator, which hands out already > defined/initialized memory (mmap of a POSIX shared memo

[Rd] custom allocators, Valgrind and uninitialized memory

2021-03-26 Thread Andreas Kersting
Hi, In my package bettermc, I use a custom allocator, which hands out already defined/initialized memory (mmap of a POSIX shared memory object). If my code is run in R which was configured --with-valgrind-instrumentation > 0, Valgrind will (correctly) complain about uninitialized memory being

Re: [Rd] Specifying C Standard in Package's Makevars File

2020-10-02 Thread Andreas Kersting
some unstated value of 'work' ... > >> * checking compilation flags in Makevars ... WARNING >> Non-portable flags in variable 'PKG_CFLAGS': >>-std=gnu11 >> > (Same for -std=c11.) >> > Thanks! Regards, >> Andreas Kersting > > Those flags are not portabl

[Rd] Specifying C Standard in Package's Makevars File

2020-09-28 Thread Andreas Kersting
(Same for -std=c11.) Thanks! Regards, Andreas Kersting __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Error in close.connection(p) : ignoring SIGPIPE signal

2019-12-05 Thread Andreas Kersting
Hi Benjamin, you cannot pipe to echo, since it does not read from stdin. echo just echos is first arg, i.e. echo /dev/stdin > /dev/null will echo the string "/dev/stdin"to /dev/stdout, which is redirected to /dev/null. Try p <- pipe("cat > /dev/null", open = "w") instead. Regards, Andreas

Re: [Rd] error in parallel:::sendMaster

2019-12-05 Thread Andreas Kersting
with this commit: https://github.com/wch/r-source/commit/e08cffac1c5b9015a1625938d568b648eb1d8aee Regards, Andreas 2019-11-28 13:54 GMT+01:00 Andreas Kersting: > Hi Tomas, > > I rebuild R (v3.5.2 for now, R-devel to follow) from the Debian package with > MC_DEBUG defined and ho

Re: [Rd] error in parallel:::sendMaster

2019-12-04 Thread Andreas Kersting
with this commit: https://github.com/wch/r-source/commit/e08cffac1c5b9015a1625938d568b648eb1d8aee Regards, Andreas 2019-11-28 13:54 GMT+01:00 Andreas Kersting: > Hi Tomas, > > I rebuild R (v3.5.2 for now, R-devel to follow) from the Debian package with > MC_DEBUG defined and ho

Re: [Rd] error in parallel:::sendMaster

2019-11-28 Thread Andreas Kersting
ven in your current debugging setup via breakpoints and=20 > backtraces, but I think it may be easier to build from source with these = > > debugging messages. > > Also if you could send me a complete example I could run that causes=20 > this on your system, that would be nice (e

Re: [Rd] error in parallel:::sendMaster

2019-11-27 Thread Andreas Kersting
he code in R around the writing I am not seeing any bug > there. If you choose to debug o n your system I can help with the instrumentation. > > Best > Tomas > > On 11/27/19 12:40 PM, Andreas Kersting wrote: >> Hi again, >> >> One important correction of

Re: [Rd] error in parallel:::sendMaster

2019-11-27 Thread Andreas Kersting
session via Rscript) I only process the second input file it will work. The other observations on R vs Rscript, NFS share etc. still hold. Sorry for this! Regards, Andreas 2019-11-27 12:10 GMT+01:00 Andreas Kersting: > Hi, > > I am facing a very weird problem with parallel::mclappl

[Rd] error in parallel:::sendMaster

2019-11-27 Thread Andreas Kersting
Hi, I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no

[Rd] make running on.exit expr uninterruptible

2019-05-22 Thread Andreas Kersting
Hi, Is there currently any way to guarantee that on.exit does not fail to execute the recorded expression because of a user interrupt arriving during function exit? Consider: f <- function() { suspendInterrupts({ on.exit(suspendInterrupts(cntr_on.exit <<- cntr_on.exit + 1L)) cntr_f

Re: [Rd] [External] most robust way to call R API functions from a secondary thread

2019-05-21 Thread Andreas Kersting
> skips all non-illustrative checks of return values). Additionally, >> R_CheckUserInterrupt() is used in combination with R_UnwindProtect() to >> regularly check for interrupts from the main thread, while still being able >> to cleanly cancel the threads before fun_running_in_main_thre

Re: [Rd] most robust way to call R API functions from a secondary thread

2019-05-21 Thread Andreas Kersting
your purposes? >> >> Do you want to parallelize your code to achieve better performance? Even >> with your proposed solution, you need synchronization and chances are that >> excessive synchronization will severely affect the expected performance >> benefi

[Rd] most robust way to call R API functions from a secondary thread

2019-05-19 Thread Andreas Kersting
hread() is left via a longjmp. This is e.g. required if the secondary threads use memory which was allocated in fun_running_in_main_thread() using e.g. R_alloc(). Best regards, Andreas Kersting #include #include #include #include extern uintptr_t R_CStackLimit; extern int R_PPS

Re: [Rd] utils::install.packages with quiet=TRUE fails for source packages on Windows

2018-01-26 Thread Andreas Kersting
I have filed a bug report here: https://github.com/rstudio/rstudio/issues/2070 Original Message From: peter dalgaard [mailto:pda...@gmail.com] Sent: Friday, Jan 26, 2018 10:15 AM GMT To: Andreas Kersting Cc: r-devel@r-project.org Subject: [Rd] utils::install.packages

Re: [Rd] utils::install.packages with quiet=TRUE fails for source packages on Windows

2018-01-26 Thread Andreas Kersting
Just noticed that this problem only occurs from within RStudio (v1.1.414). Any ideas why? Am 26.01.2018 um 08:56 schrieb Andreas Kersting: Hi, Installing a source package on Windows using utils::install.packages() with quiet=TRUE fails, while it works with the default quiet = FALSE

[Rd] utils::install.packages with quiet=TRUE fails for source packages on Windows

2018-01-25 Thread Andreas Kersting
Hi, Installing a source package on Windows using utils::install.packages() with quiet=TRUE fails, while it works with the default quiet = FALSE. The problem seems to be caused by the fact that when quiet = TRUE, stdout and stderr are set to FALSE when calling "R CMD INSTALL" with

Re: [Rd] [WISH / PATCH] possibility to split string literals across multiple lines

2017-06-14 Thread Andreas Kersting
Original Message From: Hadley Wickham [mailto:h.wick...@gmail.com] Sent: Wednesday, Jun 14, 2017 2:51 PM GMT To: Simon Urbanek Cc: Andreas Kersting; r-devel@r-project.org Subject: [Rd] [WISH / PATCH] possibility to split string literals across multiple lines On Wed, Jun 14

Re: [Rd] [WISH / PATCH] possibility to split string literals across multiple lines

2017-06-14 Thread Andreas Kersting
Original Message From: Duncan Murdoch [mailto:murdoch.dun...@gmail.com] Sent: Wednesday, Jun 14, 2017 1:36 PM GMT To: Andreas Kersting Cc: r-devel Subject: [Rd] [WISH / PATCH] possibility to split string literals across multiple lines On 14/06/2017 6:45 AM, Andreas Kersting

Re: [Rd] [WISH / PATCH] possibility to split string literals across multiple lines

2017-06-14 Thread Andreas Kersting
On Wed, 14 Jun 2017 06:12:09 -0500, Duncan Murdoch <murdoch.dun...@gmail.com> wrote: > On 14/06/2017 5:58 AM, Andreas Kersting wrote: > > Hi, > > > > I would really like to have a way to split long string literals across > > multiple lines in R. > > I don'

[Rd] [WISH / PATCH] possibility to split string literals across multiple lines

2017-06-14 Thread Andreas Kersting
Hi, I would really like to have a way to split long string literals across multiple lines in R. Currently, if a string literal spans multiple lines, there is no way to inhibit the introduction of newline characters: > "aaa + bbb" [1] "aaa\nbbb" If a line ends with a backslash, it is just

[Rd] print for lists evaluates "AsIs"-elements of type "language"

2017-06-13 Thread Andreas Kersting
Consider the following code snippets: > list(quote(1 + 1), I(quote(1 + 1))) [[1]] 1 + 1 [[2]] [1] 2 # should also be 1 + 1!? > str(list(quote(1 + 1), I(quote(1 + 1 List of 2 $ : language 1 + 1 $ : language + 1 1 # why is this line different from the one above? ..- attr(*, "class")=

Re: [Rd] Missing objects using dump.frames for post-mortem debugging of crashed batch jobs. Bug or gap in documentation?

2016-11-28 Thread Andreas Kersting
e = FALSE) in an interactive > > session debugging works as expected because it creates a > > global variable called "last.dump" and the workspace is > > still loaded. > > > In the batch job scenario however the workspace is NOT > > saved