Hi Ramos, Thank you very much – this is really informative. Can I ask you how you got the tracback? I’m not really used at debugging this sort of issues.
I can see that the EpiMix::EpiMix function we use does use foreach (foreach::makeCluster and %dopar%). I have no experience with foreach, but I had a dig, and I can see that this function cleans up the parallel environment with stopCluster only before just one of the several returns it has. So it is possible that the environment isn’t cleaned up between consequent function calls. We will investigate. Of course if anyone with more foreach experience wants to pinch in, they are very welcome Thank you again Matteo Tiberti Danish Cancer Institute Strandboulevarden 49 DK-2100 Copenhagen Telephone: +45 35 25 73 07 – a part of the Danish Cancer Society [An image showing the Danish Cancer Society's name and logo called the bow in red font. Displayed in red]<https://www.cancer.dk/?utm_source=email&utm_medium=medarbejderemail&utm_campaign=medarbejderemail&utm_content=cancerdk> www.cancer.dk<https://www.cancer.dk/international/> | Our privacy policy<https://www.cancer.dk/om-os/privatlivspolitik/> From: Ramos Perez, Marcel <marcel.ramospe...@roswellpark.org> Date: Friday, 15 March 2024 at 22.44 To: bioc-devel@r-project.org <bioc-devel@r-project.org> Cc: Matteo Tiberti <tibe...@cancer.dk> Subject: Re: EXTERNAL: [Bioc-devel] "invalid connection" when building vignette - Windows only Hi Matteo, I was also able to reproduce on Bioc 3.18 on Windows with a traceback (see below). It may be that there are symbol clashes between parallel runs in the `foreach:::.foreachGlobals` environment as described in https://stackoverflow.com/a/67497500<https://stackoverflow.com/a/67497500> Clearing the symbols in that environment alleviates the problem temporarily until running the chunk in line 1124 where the same error occurs. Someone with more experience using `foreach` and socket clusters may be able to help. Note that the code should be running on either one or two cores by default. Best regards, Marcel ``` processing file: Moonlight2R.Rmd |....................... | 45% [unnamed-chunk-20] Quitting from lines 727-746 [unnamed-chunk-20] (Moonlight2R.Rmd) Error in `summary.connection()`: ! invalid connection Backtrace: 1. Moonlight2R::GMA(...) 2. EpiMix::EpiMix(...) at Moonlight2R/R/GMA.R:117:3 3. EpiMix:::MethylMix_MixtureModel(...) 4. ... %dopar% ... ... 7. parallel:::sendCall(cl[[i]], fun, list(...)) 8. parallel:::postNode(...) 10. parallel:::sendData.SOCKnode(...) 11. base::serialize(data, node$con) 13. base::summary.connection(connection) Execution halted ``` On 3/15/24 6:05 AM, Matteo Tiberti via Bioc-devel wrote: Dear developers, we've been getting a build fail for our Moonlight2R package on Windows only, during vignette build in CHECK, which just prints out an "invalid connection" error: Quitting from lines 727-746 [unnamed-chunk-20] (Moonlight2R.Rmd) Error: processing vignette 'Moonlight2R.Rmd' failed with diagnostics: invalid connection --- failed re-building 'Moonlight2R.Rmd' The following snippet is the code chunk that causes the fail: data("dataMethyl") data("dataFilt") data("dataPRA") data("DEGsmatrix") data("LUAD_sample_anno") data("NCG") data("EncodePromoters") data("MetEvidenceDriver") # Subset column names (sample names) in expression data to patient level pattern <- "^(.{4}-.{2}-.{4}-.{2}).*" colnames(dataFilt) <- sub(pattern, "\\1", colnames(dataFilt)) dataGMA <- GMA(dataMET = dataMethyl, dataEXP = dataFilt, dataPRA = dataPRA, dataDEGs = DEGsmatrix, sample_info = LUAD_sample_anno, met_platform = "HM450", prevalence_filter = NULL, output_dir = "./GMAresults", cores = 1, roadmap.epigenome.ids = "E096", roadmap.epigenome.groups = NULL) I could reproduce this fail on my local Windows machine (on release rather than devel at the moment), so it's probably not an issue with the builders. It builds and runs successfully on Linux, both on devel and release. I have also tried running the same code by hand on the R prompt, and that completes successfully. The error I get when trying to build the vignette manually is possibly more informative: devtools::build_vignettes('../Downloads/Moonlight2R-devel/Moonlight2R-devel/') ℹ Installing Moonlight2R in temporary library ℹ Building vignettes for Moonlight2R --- re-building 'Moonlight2R.Rmd' using rmarkdown processing file: Moonlight2R.Rmd |....................... | 45% [unnamed-chunk-20]trying URL 'https://bioconductor.org/packages/3.18/data/experiment/src/contrib/sesameData_1.20.0.tar.gz<https://bioconductor.org/packages/3.18/data/experiment/src/contrib/sesameData_1.20.0.tar.gz>' trying URL 'https://bioconductor.org/packages/3.18/data/experiment/src/contrib/sesameData_1.20.0.tar.gz<https://bioconductor.org/packages/3.18/data/experiment/src/contrib/sesameData_1.20.0.tar.gz>' Content type 'application/x-gzip' length 1975798 bytes (1.9 MB) ================================================== downloaded 1.9 MB * installing *source* package 'sesameData' ... ** using staged installation ** R ** inst ** byte-compile and prepare package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes ** testing if installed package can be loaded from temporary location ** testing if installed package can be loaded from final location ** testing if installed package keeps a record of temporary installation path * DONE (sesameData) The downloaded source packages are in 'C:\Users\Work\AppData\Local\Temp\RtmpmsVLSL\downloaded_packages' Quitting from lines 727-746 [unnamed-chunk-20] (Moonlight2R.Rmd) Error: processing vignette 'Moonlight2R.Rmd' failed with diagnostics: invalid connection --- failed re-building 'Moonlight2R.Rmd' SUMMARY: processing the following file failed: 'Moonlight2R.Rmd' so it installs the sesameData package (because of ExperimentHub::ExperimentHub()[['EH3675']]) , and then fails like that. We have very little experience in development under Windows and are a bit stumped on what might be the problem; do you know whether this error message comes from anything in particular or do you have any further idea on how to troubleshoot this? I assume the error message does not refer to connecting to some server on the internet, since all works fine on Linux and I can reproduce the problem from different machines/connections. But happy to hear your thoughts Thank you and kind regards, Matteo Tiberti Danish Cancer Institute Strandboulevarden 49 DK-2100 Copenhagen Telephone: +45 35 25 73 07 – a part of the Danish Cancer Society [An image showing the Danish Cancer Society's name and logo called the bow in red font. Displayed in red]<https://www.cancer.dk/?utm_source=email&utm_medium=medarbejderemail&utm_campaign=medarbejderemail&utm_content=cancerdk><https://www.cancer.dk/?utm_source=email&utm_medium=medarbejderemail&utm_campaign=medarbejderemail&utm_content=cancerdk> www.cancer.dk<http://www.cancer.dk><https://www.cancer.dk/international/><https://www.cancer.dk/international/> | <https://www.cancer.dk/om-os/privatlivspolitik/><https://www.cancer.dk/om-os/privatlivspolitik/> Our privacy policy<https://www.cancer.dk/om-os/privatlivspolitik/><https://www.cancer.dk/om-os/privatlivspolitik/> _______________________________________________ Bioc-devel@r-project.org<mailto:Bioc-devel@r-project.org> mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel<https://stat.ethz.ch/mailman/listinfo/bioc-devel> --- Marcel Ramos Bioconductor Core Team Roswell Park Comprehensive Cancer Center Dept. of Biostatistics & Bioinformatics Elm St. & Carlton St. Buffalo, New York 14263 This email message may contain legally privileged and/or confidential information. If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited. If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you. _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel