Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
Henrik,
I completely agree with everything you wrote, but note that the issue at hand
is using `:::' in *the same* package, for example when a package needs to
access its own internal functions from an outside context, where running on a
cluster node set up by the package is one thing I can think of. So there is no
API contract to violate, except the one the package makes with itself. Given
this, I'm inclined to agree with David: the language provides an obvious way to
do this, why write a semantic kludge that is obviously intended only to
circumvent the CRAN warning to achieve something that is necessary for the
package? Of course, just my €.02 in a thought-provoking discussion!
Cesko
Op 14-09-2020 om 21:42 schreef Henrik Bengtsson:
Without having read all of the comments already made here, but my
understanding why ::: is not allowed is because you are reaching into
the internal API that the package owner does not guarantee will exist
in the next release. If you rely on the internal code of another CRAN
package in your CRAN package, your CRAN package might break without
your control. This might release an avalanche of reverse package
dependencies failing on CRAN.
The only thing you can safely rely on is the API that is explicitly
*exported* by an R package. In order for the maintainer to break that
API for reverse dependent packages, they need to go through a process
of deprecating and defuncting what they want to break/remove - a
process that involves multiple releases and often reaching out to
package maintainers and asking them to update accordingly. CRAN runs
reverse package dependency checks making sure that a package does not
break its exported API. If it does, it will not roll out on CRAN.
So, in that sense CRAN helps uphold the contract of the exported APIs.
In contrast, a maintainer can do whatever they want whenever they want
with their internal code/API.
With more and more packages being infrastructure packages, I think
there is room for "protected" API, which is not exported to avoid
cluttering up the search path for end-users while it yet provides a
contract toward package developers relying on it. There are various
ways to emulate such protected APIs but we don't have a standard and
there's a risk that 'R CMD check' fails to detect when the contract is
broken (resulting in delayed run-time errors on the user end).
My $.02
Henrik
On Mon, Sep 14, 2020 at 12:06 PM David Kepplinger
wrote:
Yes, my view is certainly rigid and I agree that in the cases where the
function is actually used directly by the user, exporting it is the correct
step.
However, it seems some packages actually need to access internal functions
from an outside context, but the code that accesses the function is
logically contained completely inside the package. In these cases, package
maintainers seem to be looking for alternatives to `:::` for the sake of
avoiding the R CMD check note. I argue that the work arounds, however,
either (a) achieve the exact same result as `:::`, but in a less
transparent and likely more error prone way, or (b) unnecessarily making an
internal function available to the user.
I also agree with the CRAN team that package maintainers need to be made
aware of the issue when using `:::` inside their package as it is most
likely unnecessary. But the phrasing of the note ("almost never needs to
use :::") combined with a lack of transparent guidelines on when it is
acceptable leads to maintainers looking for alternatives mimicking the
behavior of `:::`. I haven't found any official instructions in Writing R
extensions or on the mailing list under what circumstances `:::` is deemed
to be acceptable by the CRAN team (I have to admit searching for `:::` in
the archives yields so many results I haven't looked at all of them). It's
probably impossible to conceive every possible use case for `:::`, but a
good start may be to have something in the documentation explicitly
mentioning commonly observed patterns where `:::` is not acceptable, and
the common exceptions to the rule (if there are any).
Maybe this issue is so miniscule and almost never comes up that it's not
worth mentioning in the documentation.
Best,
David
On Mon, Sep 14, 2020 at 3:19 AM Georgi Boshnakov <
georgi.boshna...@manchester.ac.uk> wrote:
You may have a case to argue to CRAN that you can get the "almost"
exemption (can't say without details) but your views look overly rigid.
Exporting an object and marking it as internal is not a "work around",
even less a "dirty trick".
Export makes the object available outside the package's namespace and
makes it clear that this is intentional.
If you can't drop the 'package:::' prefix in your use case, this means
that this is what you actually do (i.e. use those objects outside the
namespace of the package). I would be grateful to CRAN for asking me to
export and hence document this.
Georgi Boshnakov
PS Note that there is no such thing as "public namespace".
Re: [R-pkg-devel] Etiquette for package submissions that do not automatically pass checks?
Duncan, Joshua,
Thanks for the feedback. I had indeed forgotten to increment the version
number, so that may explain it. I'll give it until a few days after the 24th,
and then just resubmit with a new version number.
Also, thanks for introducing me to the foghorn package, I wasn't aware of it
before.
Best,
Cesko
Op 14-08-2020 om 22:31 schreef Joshua Ulrich:
On Fri, Aug 14, 2020 at 2:54 PM Duncan Murdoch wrote:
On 14/08/2020 3:08 p.m., Cesko Voeten wrote:
A while ago, I submitted an update to my package 'buildmer' that does not pass
R CMD check. This is deliberate. The package contains functionality to run on
cluster nodes that were set up by the user and needs to access its own internal
functions from there. In previous versions of the package, I had maintained a
list of those functions and clusterExport()ed them, but that had the side
effect of overwriting any same-named user objects on the user-provided cluster
nodes, which I thought was poor form. The update therefore accesses these
functions using ':::', which triggers a check warning.
I thought the etiquette was to explain this in the 'Comments' box when
submitting, but this gave me the same automated message that the package does
not pass checks and that I should fix it or reply-all and explain. This led me
to believe that I should not have used the 'Comments' box for this purpose,
hence I resubmitted the package leaving the comments box empty, and I
replied-all to the subsequent e-mail I got with an explanation similar to the
above.
It seems to me that what you should have done is "reply-all and
explain", as the automated message said.
It has now been a while since I sent that e-mail (ten days), and I have yet to
hear back. I was wondering if the message had gotten lost, if they simply
haven't gotten around to it yet (I have no idea how much mail they receive on a
daily basis, but I'd think it's a lot more than I do), or if I should have
handled this differently. Only CRAN can answer the first two questions, but
before I bother them: was this the correct procedure, or should I simply have
done something differently?
You can see the state of your submission using the foghorn package.
cran_incoming("buildmer") currently shows your package is in the
"archive", which means "package rejected: it does not pass the checks
cleanly and the problems are unlikely to be false positives".
I only see version 1.7 there, which may indicate that you resubmitted
exactly the same package (down to the version number). As the
instructions at
https://cran.r-project.org/web/packages/policies.html#Re_002dsubmission
say, "Increasing the version number at each submission reduces confusion
so is preferred even when a previous submission was not accepted."
What I'd suggest now is that you do nothing more for a day or two,
because CRAN members who aren't on holiday might read and respond to
your message. If you don't hear anything, then I'd start over again,
with a new version number, and an explanation in the comments, and
likely a followup reply-all.
You have more than a few days. As it says on CRAN:
"CRAN submission is offline from Aug 14 to Aug 24, 2020 (CRAN team
vacation and maintainance work)"
Alternatively, you could export those troublesome functions from your
package but document them as for internal use only. Renaming them with
a name starting with "." will make them harder for users to stumble
upon, but you can still access them using buildmer::.something, you
shouldn't need clusterExport(). Then you will meet the technical
requirement and not need any explanation.
Duncan Murdoch
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] Etiquette for package submissions that do not automatically pass checks?
A while ago, I submitted an update to my package 'buildmer' that does not pass R CMD check. This is deliberate. The package contains functionality to run on cluster nodes that were set up by the user and needs to access its own internal functions from there. In previous versions of the package, I had maintained a list of those functions and clusterExport()ed them, but that had the side effect of overwriting any same-named user objects on the user-provided cluster nodes, which I thought was poor form. The update therefore accesses these functions using ':::', which triggers a check warning. I thought the etiquette was to explain this in the 'Comments' box when submitting, but this gave me the same automated message that the package does not pass checks and that I should fix it or reply-all and explain. This led me to believe that I should not have used the 'Comments' box for this purpose, hence I resubmitted the package leaving the comments box empty, and I replied-all to the subsequent e-mail I got with an explanation similar to the above. It has now been a while since I sent that e-mail (ten days), and I have yet to hear back. I was wondering if the message had gotten lost, if they simply haven't gotten around to it yet (I have no idea how much mail they receive on a daily basis, but I'd think it's a lot more than I do), or if I should have handled this differently. Only CRAN can answer the first two questions, but before I bother them: was this the correct procedure, or should I simply have done something differently? Thanks, Cesko __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Seeking for best way to manage package transitive dependencies
Package B should import only the packages and functions that are used by package B. If package B does not use functions from package C, package B should not import package C. What package A does is package A's problem, not package B's. If package A requires package C, install.packages() will automatically install it when package C is being installed, just as package C is installed when package B is being installed. I submit the following thought experiment: suppose that, at a later point in time, package A is re-written to not need package C. In this case, it would be incorrect for package B to still needlessly import package C. Best, Cesko Op 10-10-2019 om 17:14 schreef neonira Arinoem: Suppose package B imports package A and package A imports package C. Shall package B import package C, knowing that package B will use functions from package A that are using functions from package C ? Currently, package B imports package C. This leads to a note from CRAN stating Namespace in Imports field not imported from: ‘lubridate’ All declared Imports should be used. Doing so, I expect package B user not to worry about needed package B dependencies. What is the best way to to manage package transitive dependencies, in such a case ? Neonira. [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
