Re: [R-pkg-devel] is R CMD build --compact-vignettes working as expected?
Dear all, First, thank you to everyone who responded. Let me be more specific about what I did, focusing on macOS (since I did most of this on Windows too): (1) I observed the problem with the Rcmdr package, including version 2.7-1, which is newly on CRAN. Presumably, it wouldn't be hard to duplicate what I did. (2) I previously tried both --compact-vignettes="both", --compact-vignettes=both, and also just --compact-vignettes. R CMD build says that it's "* compacting vignettes and other PDF files", and R CMD check --as-cran doesn't complain, reporting "* checking sizes of PDF files under ‘inst/doc’ ... OK." (3) Nothing changes when, following Duncan's advice, I set GS_QUALITY=screen In the tarball built by R CMD build, the vignette in inst/doc/Getting-Started-with-the-Rcmdr.pdf is 1.6 MB. When I compacted the vignette with Adobe Acrobat, it was reduced to 395 KB. Putting all of this together, I suspect that Acrobat may just do a better job than gs and qpdf. Best, John On 2020-10-08 12:06 p.m., Duncan Murdoch wrote: When I tried that on MacOS, it did the gs compression with gs_quality set to "none", which does nothing. I don't know what quality CRAN uses, but for me setting the environment variable GS_QUALITY=screen made a big difference. Duncan Murdoch On 08/10/2020 11:10 a.m., John Fox wrote: Dear Ben, Actually, what I used was --compact-vignettes="both", with qpdf and gs installed on my Windows and Mac machines, and that apparently didn't work for me. Best, John On 2020-10-07 10:06 p.m., Ben Bolker wrote: OK, I think I see the problem. tl;dr use --compact-vignettes="both" when building the vignettes. --compact-vignettes by default only tries qpdf. when the R CMD check --as-cran is run, it tries both qpdf and gs. Since gs (apparently, in this case) compresses more aggressively than qpdf, it succeeds in compressing further, and the check complains. From R CMD build --help: --compact-vignettes= try to compact PDF files under inst/doc: "no" (default), "qpdf", "gs", "gs+qpdf", "both" --compact-vignettes same as --compact-vignettes=qpdf I don't know if this is worth documenting somewhere, or modifying the behaviour to make "both" the default for --compact-vignettes ? On 10/7/20 8:35 PM, Duncan Murdoch wrote: On 07/10/2020 8:32 p.m., Ben Bolker wrote: Thanks for the tip, I'll take a look. Given that three relatively experienced package authors all seem to have experienced similar issues, it seems that maybe this is worth figuring out/maybe I'm not just doing something boneheaded. Just to clarify: I've never noticed the problem you mention. I just know how to debug R CMD build. Duncan cheers Ben On 10/7/20 8:31 PM, Duncan Murdoch wrote: I don't know the answer to your question, but you can debug the --compact-vignettes option as follows. debug(tools::compactPDF) tools:::.build_packages(c("--compact-vignettes", "pkgdir")) where "pkgdir" is the directory of the source of your package. Add extra options to the build as separate elements of the argument to .build(): this function is called after args have been parsed. When I do that, I see that it rejects the compaction, because none of mine benefit much from it: it wants at least a 10% and 10K reduction. But Ben's example met those criteria. When I trick it into accepting the compaction, it does put the compacted PDF into the tarball. Duncan Murdoch On 07/10/2020 6:03 p.m., John Fox wrote: Dear Ben, On 2020-10-07 5:26 p.m., Ben Bolker wrote: I hope so too. The (annoying) workaround is to compact the vignette yourself (using qpdf directly or using tools::compactPDF), then use no-build-vignettes. The problem there is whatever's supposed to happen with building vignette indices. The ugly workaround, I guess, is to build the tarball, compact the vignettes oneself, then *replace* them in the tarball. (Obviously I can automate that, but it seems as though it would be unnecessary if I knew what was going on ...) I've used both of these workarounds and agree that it would be nice to avoid them. After all, what is the --compact-vignettes argument for? Best, John cheers Ben On 10/7/20 4:10 PM, John Fox wrote: Dear Ben, I was hoping that someone would pick up on this problem, because I've experienced the same issue of --compact-vignettes apparently ignored, e.g., with the Rcmdr package under R 4.0.2 on both macOS and Windows. Best, John John Fox, Professor Emeritus McMaster University Hamilton, Ontario, Canada web: https://socialsciences.mcmaster.ca/jfox/ On 2020-10-05 1:09 p.m., Ben Bolker wrote: Am I confused, or doing something wrong, or ... ? I have qpdf installed, and am running R CMD build with --compact-vignettes, but the PDF in the tarball doesn't seem to be compressed despite the fact that the output messages say "compacting vignettes ..."
Re: [R-pkg-devel] is R CMD build --compact-vignettes working as expected?
When I tried that on MacOS, it did the gs compression with gs_quality set to "none", which does nothing. I don't know what quality CRAN uses, but for me setting the environment variable GS_QUALITY=screen made a big difference. Duncan Murdoch On 08/10/2020 11:10 a.m., John Fox wrote: Dear Ben, Actually, what I used was --compact-vignettes="both", with qpdf and gs installed on my Windows and Mac machines, and that apparently didn't work for me. Best, John On 2020-10-07 10:06 p.m., Ben Bolker wrote: OK, I think I see the problem. tl;dr use --compact-vignettes="both" when building the vignettes. --compact-vignettes by default only tries qpdf. when the R CMD check --as-cran is run, it tries both qpdf and gs. Since gs (apparently, in this case) compresses more aggressively than qpdf, it succeeds in compressing further, and the check complains. From R CMD build --help: --compact-vignettes= try to compact PDF files under inst/doc: "no" (default), "qpdf", "gs", "gs+qpdf", "both" --compact-vignettes same as --compact-vignettes=qpdf I don't know if this is worth documenting somewhere, or modifying the behaviour to make "both" the default for --compact-vignettes ? On 10/7/20 8:35 PM, Duncan Murdoch wrote: On 07/10/2020 8:32 p.m., Ben Bolker wrote: Thanks for the tip, I'll take a look. Given that three relatively experienced package authors all seem to have experienced similar issues, it seems that maybe this is worth figuring out/maybe I'm not just doing something boneheaded. Just to clarify: I've never noticed the problem you mention. I just know how to debug R CMD build. Duncan cheers Ben On 10/7/20 8:31 PM, Duncan Murdoch wrote: I don't know the answer to your question, but you can debug the --compact-vignettes option as follows. debug(tools::compactPDF) tools:::.build_packages(c("--compact-vignettes", "pkgdir")) where "pkgdir" is the directory of the source of your package. Add extra options to the build as separate elements of the argument to .build(): this function is called after args have been parsed. When I do that, I see that it rejects the compaction, because none of mine benefit much from it: it wants at least a 10% and 10K reduction. But Ben's example met those criteria. When I trick it into accepting the compaction, it does put the compacted PDF into the tarball. Duncan Murdoch On 07/10/2020 6:03 p.m., John Fox wrote: Dear Ben, On 2020-10-07 5:26 p.m., Ben Bolker wrote: I hope so too. The (annoying) workaround is to compact the vignette yourself (using qpdf directly or using tools::compactPDF), then use no-build-vignettes. The problem there is whatever's supposed to happen with building vignette indices. The ugly workaround, I guess, is to build the tarball, compact the vignettes oneself, then *replace* them in the tarball. (Obviously I can automate that, but it seems as though it would be unnecessary if I knew what was going on ...) I've used both of these workarounds and agree that it would be nice to avoid them. After all, what is the --compact-vignettes argument for? Best, John cheers Ben On 10/7/20 4:10 PM, John Fox wrote: Dear Ben, I was hoping that someone would pick up on this problem, because I've experienced the same issue of --compact-vignettes apparently ignored, e.g., with the Rcmdr package under R 4.0.2 on both macOS and Windows. Best, John John Fox, Professor Emeritus McMaster University Hamilton, Ontario, Canada web: https://socialsciences.mcmaster.ca/jfox/ On 2020-10-05 1:09 p.m., Ben Bolker wrote: Am I confused, or doing something wrong, or ... ? I have qpdf installed, and am running R CMD build with --compact-vignettes, but the PDF in the tarball doesn't seem to be compressed despite the fact that the output messages say "compacting vignettes ..." $ R CMD build --compact-vignettes lme4 * checking for file ‘lme4/DESCRIPTION’ ... OK * preparing ‘lme4’: * checking DESCRIPTION meta-information ... OK * cleaning src * installing the package to process help pages * saving partial Rd database * creating vignettes ... OK Warning: ‘inst/doc’ files ‘lmerperf.html’, ‘lmer.pdf’, ‘PLSvGLS.pdf’, ‘Theory.pdf’ ignored as vignettes have been rebuilt. Run R CMD build with --no-build-vignettes to prevent rebuilding. * compacting vignettes and other PDF files * cleaning src * checking for LF line-endings in source and make files and shell scripts * checking for empty or unneeded directories * building ‘lme4_1.1-24.tar.gz’ The copy of lmer.pdf in the resulting tarball is 900K or so: $ tar ztvf lme4_1.1-24.tar.gz lme4/inst/doc/lmer.pdf -rw-r--r-- bolker/bolker 907022 2020-10-05 12:59 lme4/inst/doc/lmer.pdf The previously built (and manually compacted) version of lmer.pdf in the tarball is 500K: $ ls -l lme4/inst/doc/lmer.pdf -rw-r--r-- 1 bolker bolker 495199 Oct 3 22:15
Re: [R-pkg-devel] is R CMD build --compact-vignettes working as expected?
Hi both, I am not sure if this might be the issue here, but I think on both Linux and Windows I need to use the both option without quotation marks: --compact-vignettes=both Best, Henrik Am Do., 8. Okt. 2020 um 17:21 Uhr schrieb Ben Bolker : > >Huh, weird. "both" worked for me. > >My debugging process involved instrumenting the compression function > to report what it was doing (see below), rebuilding R, and running the > build and check --as-cran processes again. (If the package you're > experiencing problems with is public, feel free to point me to it and > I'll try to take a look.) > >cheers > Ben > > Index: src/library/tools/R/admin.R > === > --- src/library/tools/R/admin.R (revision 79311) > +++ src/library/tools/R/admin.R (working copy) > @@ -1088,6 +1088,7 @@ > } > if(!res && file.exists(tf)) { > old <- file.size(p); new <- file.size(tf) > +cat(sprintf("compression: old=%f, new=%f, new/old=%f, > old-new=%f",old, new, new/old,old-new),"\n") > if(new/old < 0.9 && new < old - 1e4) { > file.copy(tf, p, overwrite = TRUE) > ans[p, ] <- c(old, new) > > > On 10/8/20 11:10 AM, John Fox wrote: > > Dear Ben, > > > > Actually, what I used was --compact-vignettes="both", with qpdf and gs > > installed on my Windows and Mac machines, and that apparently didn't > > work for me. > > > > Best, > > John > > > > On 2020-10-07 10:06 p.m., Ben Bolker wrote: > >>OK, I think I see the problem. tl;dr use --compact-vignettes="both" > >> when building the vignettes. > >> > >>--compact-vignettes by default only tries qpdf. > >> > >>when the R CMD check --as-cran is run, it tries both qpdf and gs. > >> Since gs (apparently, in this case) compresses more aggressively than > >> qpdf, it succeeds in compressing further, and the check complains. > >> > >> From R CMD build --help: > >> > >> --compact-vignettes= try to compact PDF files under inst/doc: > >> "no" (default), "qpdf", "gs", "gs+qpdf", "both" > >>--compact-vignettes same as --compact-vignettes=qpdf > >> > >> > >>I don't know if this is worth documenting somewhere, or modifying > >> the behaviour to make "both" the default for --compact-vignettes ? > >> > >> > >> On 10/7/20 8:35 PM, Duncan Murdoch wrote: > >>> On 07/10/2020 8:32 p.m., Ben Bolker wrote: > Thanks for the tip, I'll take a look. > > Given that three relatively experienced package authors all seem to > have experienced similar issues, it seems that maybe this is worth > figuring out/maybe I'm not just doing something boneheaded. > >>> > >>> Just to clarify: I've never noticed the problem you mention. I just > >>> know how to debug R CMD build. > >>> > >>> Duncan > >>> > > cheers > Ben > > On 10/7/20 8:31 PM, Duncan Murdoch wrote: > > I don't know the answer to your question, but you can debug the > > --compact-vignettes option as follows. > > > > debug(tools::compactPDF) > > tools:::.build_packages(c("--compact-vignettes", "pkgdir")) > > > > where "pkgdir" is the directory of the source of your package. Add > > extra options to the build as separate elements of the argument to > > .build(): this function is called after args have been parsed. > > > > When I do that, I see that it rejects the compaction, because none of > > mine benefit much from it: it wants at least a 10% and 10K reduction. > > But Ben's example met those criteria. When I trick it into accepting > > the compaction, it does put the compacted PDF into the tarball. > > > > Duncan Murdoch > > > > On 07/10/2020 6:03 p.m., John Fox wrote: > >> Dear Ben, > >> > >> On 2020-10-07 5:26 p.m., Ben Bolker wrote: > >>> I hope so too. The (annoying) workaround is to compact the > >>> vignette > >>> yourself (using qpdf directly or using tools::compactPDF), then use > >>> no-build-vignettes. The problem there is whatever's supposed to > >>> happen > >>> with building vignette indices. The ugly workaround, I > >>> guess, is to > >>> build the tarball, compact the vignettes oneself, then *replace* > >>> them in > >>> the tarball. (Obviously I can automate that, but it seems as > >>> though it > >>> would be unnecessary if I knew what was going on ...) > >> > >> I've used both of these workarounds and agree that it would be > >> nice to > >> avoid them. After all, what is the --compact-vignettes argument for? > >> > >> Best, > >> John > >> > >>> > >>> cheers > >>> Ben > >>> > >>> > >>> On 10/7/20 4:10 PM, John Fox wrote: > Dear Ben, > > I was hoping that someone would pick up on this problem, because > I've >
Re: [R-pkg-devel] is R CMD build --compact-vignettes working as expected?
Dear Ben, Actually, what I used was --compact-vignettes="both", with qpdf and gs installed on my Windows and Mac machines, and that apparently didn't work for me. Best, John On 2020-10-07 10:06 p.m., Ben Bolker wrote: OK, I think I see the problem. tl;dr use --compact-vignettes="both" when building the vignettes. --compact-vignettes by default only tries qpdf. when the R CMD check --as-cran is run, it tries both qpdf and gs. Since gs (apparently, in this case) compresses more aggressively than qpdf, it succeeds in compressing further, and the check complains. From R CMD build --help: --compact-vignettes= try to compact PDF files under inst/doc: "no" (default), "qpdf", "gs", "gs+qpdf", "both" --compact-vignettes same as --compact-vignettes=qpdf I don't know if this is worth documenting somewhere, or modifying the behaviour to make "both" the default for --compact-vignettes ? On 10/7/20 8:35 PM, Duncan Murdoch wrote: On 07/10/2020 8:32 p.m., Ben Bolker wrote: Thanks for the tip, I'll take a look. Given that three relatively experienced package authors all seem to have experienced similar issues, it seems that maybe this is worth figuring out/maybe I'm not just doing something boneheaded. Just to clarify: I've never noticed the problem you mention. I just know how to debug R CMD build. Duncan cheers Ben On 10/7/20 8:31 PM, Duncan Murdoch wrote: I don't know the answer to your question, but you can debug the --compact-vignettes option as follows. debug(tools::compactPDF) tools:::.build_packages(c("--compact-vignettes", "pkgdir")) where "pkgdir" is the directory of the source of your package. Add extra options to the build as separate elements of the argument to .build(): this function is called after args have been parsed. When I do that, I see that it rejects the compaction, because none of mine benefit much from it: it wants at least a 10% and 10K reduction. But Ben's example met those criteria. When I trick it into accepting the compaction, it does put the compacted PDF into the tarball. Duncan Murdoch On 07/10/2020 6:03 p.m., John Fox wrote: Dear Ben, On 2020-10-07 5:26 p.m., Ben Bolker wrote: I hope so too. The (annoying) workaround is to compact the vignette yourself (using qpdf directly or using tools::compactPDF), then use no-build-vignettes. The problem there is whatever's supposed to happen with building vignette indices. The ugly workaround, I guess, is to build the tarball, compact the vignettes oneself, then *replace* them in the tarball. (Obviously I can automate that, but it seems as though it would be unnecessary if I knew what was going on ...) I've used both of these workarounds and agree that it would be nice to avoid them. After all, what is the --compact-vignettes argument for? Best, John cheers Ben On 10/7/20 4:10 PM, John Fox wrote: Dear Ben, I was hoping that someone would pick up on this problem, because I've experienced the same issue of --compact-vignettes apparently ignored, e.g., with the Rcmdr package under R 4.0.2 on both macOS and Windows. Best, John John Fox, Professor Emeritus McMaster University Hamilton, Ontario, Canada web: https://socialsciences.mcmaster.ca/jfox/ On 2020-10-05 1:09 p.m., Ben Bolker wrote: Am I confused, or doing something wrong, or ... ? I have qpdf installed, and am running R CMD build with --compact-vignettes, but the PDF in the tarball doesn't seem to be compressed despite the fact that the output messages say "compacting vignettes ..." $ R CMD build --compact-vignettes lme4 * checking for file ‘lme4/DESCRIPTION’ ... OK * preparing ‘lme4’: * checking DESCRIPTION meta-information ... OK * cleaning src * installing the package to process help pages * saving partial Rd database * creating vignettes ... OK Warning: ‘inst/doc’ files ‘lmerperf.html’, ‘lmer.pdf’, ‘PLSvGLS.pdf’, ‘Theory.pdf’ ignored as vignettes have been rebuilt. Run R CMD build with --no-build-vignettes to prevent rebuilding. * compacting vignettes and other PDF files * cleaning src * checking for LF line-endings in source and make files and shell scripts * checking for empty or unneeded directories * building ‘lme4_1.1-24.tar.gz’ The copy of lmer.pdf in the resulting tarball is 900K or so: $ tar ztvf lme4_1.1-24.tar.gz lme4/inst/doc/lmer.pdf -rw-r--r-- bolker/bolker 907022 2020-10-05 12:59 lme4/inst/doc/lmer.pdf The previously built (and manually compacted) version of lmer.pdf in the tarball is 500K: $ ls -l lme4/inst/doc/lmer.pdf -rw-r--r-- 1 bolker bolker 495199 Oct 3 22:15 lme4/inst/doc/lmer.pdf Is 'R CMD build' confused by the presence of a pre-built PDF in the inst/doc directory? Or am I somehow mistaken about how this is supposed to work? I would just use --no-build-vignettes and submit the tarball with the previously built/compressed PDF, but