Re: [R-pkg-devel] Problems with too much testing
One of the basic principles of testing is "test interface, not implementation." Tests that violate this principle become "change detector tests" instead of "correctness tests" and essentially prevent any improvements to the code. In C++ objects have "public" and "private" methods, and the testing diktat is to test the public ones. I hit this issue a lot at Google when internal clients wrote change detector tests effectively fencing in my tools. My solution was to point out the problematic tests and ask them not to do that. The intent of R as a language is to be fast, lightweight, and flexible. In the interest of keeping it that way for users, it has grown large and cumbersome for package authors. If we try to tack on a "public/private" object layout I'm afraid that would be yet another burden on package authors. The most obvious place I see to do this is in the NAMESPACE file, which is already an ugly beast. So while I agree with Duncan's characterization of the problem, to my ears the "social solution" is the cleanest one available, if we're talking about formal mechanisms. Informally, both C++ and python use underscores in variable names to suggest private data members. You could just prefix "PRIVATE" to the name of all the data members you don't want tested, where "PRIVATE" might become "." or "_", or some other character. I would argue against using underscores, but I think I've already lost that battle. On Fri, Apr 16, 2021 at 7:53 AM J C Nash wrote: > I'm generally in accord with Duncan on this. There are inevitably > situations where general > rules don't apply. Our challenge is to find practical ways to keep the > overall workload of > all participants in the process to a minimum. > > JN > > On 2021-04-16 10:18 a.m., Duncan Murdoch wrote: > > On 16/04/2021 9:49 a.m., J C Nash wrote: > >> Another approach is to change the responsibility. > >> > >> My feeling is that tests in the TESTING package should be modifiable by > the maintainer of > >> the TESTED package, with both packages suspended if the two maintainers > cannot agree. We > >> need to be able to move forward when legacy behaviour is outdated or > just plain wrong. Or, > >> in the case that I find affects me, when improvements in iterative > schemes change iterates > >> slightly. My guess is that Duncan's example is a case in point. > >> > >> I doubt this will ever occur, as it doesn't seem to be the R way. > However, I do know that > >> improvements in methods are not going to CRAN in some cases. > > > > In the cases I've been involved with the authors of the testing package > have accepted suggested changes when I've made > > them: I think that's also part of "the R way". However, this takes > time for both of us: I need to understand what > > they are intending to test before I can suggest a change to it, and they > need to understand my change before they can > > decide if it is acceptable, or whether further changes would also be > necessary. > > > > Github helps a lot with this: if the testing package is there, I can > quickly reproduce the issue, produce a fix, and > > send it to the author, who can tweak it if I've set things up properly. > > > > For the kinds of changes you're making, I suspect relaxing a tolerance > would often be enough, though if you switch > > algorithms and record that in your results, the testing package may need > to replace reference values. I think I'd be > > uncomfortable doing that. > > > > Duncan Murdoch > > > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel > [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
I'm generally in accord with Duncan on this. There are inevitably situations where general rules don't apply. Our challenge is to find practical ways to keep the overall workload of all participants in the process to a minimum. JN On 2021-04-16 10:18 a.m., Duncan Murdoch wrote: > On 16/04/2021 9:49 a.m., J C Nash wrote: >> Another approach is to change the responsibility. >> >> My feeling is that tests in the TESTING package should be modifiable by the >> maintainer of >> the TESTED package, with both packages suspended if the two maintainers >> cannot agree. We >> need to be able to move forward when legacy behaviour is outdated or just >> plain wrong. Or, >> in the case that I find affects me, when improvements in iterative schemes >> change iterates >> slightly. My guess is that Duncan's example is a case in point. >> >> I doubt this will ever occur, as it doesn't seem to be the R way. However, I >> do know that >> improvements in methods are not going to CRAN in some cases. > > In the cases I've been involved with the authors of the testing package have > accepted suggested changes when I've made > them: I think that's also part of "the R way". However, this takes time for > both of us: I need to understand what > they are intending to test before I can suggest a change to it, and they need > to understand my change before they can > decide if it is acceptable, or whether further changes would also be > necessary. > > Github helps a lot with this: if the testing package is there, I can quickly > reproduce the issue, produce a fix, and > send it to the author, who can tweak it if I've set things up properly. > > For the kinds of changes you're making, I suspect relaxing a tolerance would > often be enough, though if you switch > algorithms and record that in your results, the testing package may need to > replace reference values. I think I'd be > uncomfortable doing that. > > Duncan Murdoch > __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
On 16/04/2021 9:49 a.m., J C Nash wrote: Another approach is to change the responsibility. My feeling is that tests in the TESTING package should be modifiable by the maintainer of the TESTED package, with both packages suspended if the two maintainers cannot agree. We need to be able to move forward when legacy behaviour is outdated or just plain wrong. Or, in the case that I find affects me, when improvements in iterative schemes change iterates slightly. My guess is that Duncan's example is a case in point. I doubt this will ever occur, as it doesn't seem to be the R way. However, I do know that improvements in methods are not going to CRAN in some cases. In the cases I've been involved with the authors of the testing package have accepted suggested changes when I've made them: I think that's also part of "the R way". However, this takes time for both of us: I need to understand what they are intending to test before I can suggest a change to it, and they need to understand my change before they can decide if it is acceptable, or whether further changes would also be necessary. Github helps a lot with this: if the testing package is there, I can quickly reproduce the issue, produce a fix, and send it to the author, who can tweak it if I've set things up properly. For the kinds of changes you're making, I suspect relaxing a tolerance would often be enough, though if you switch algorithms and record that in your results, the testing package may need to replace reference values. I think I'd be uncomfortable doing that. Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
Another approach is to change the responsibility. My feeling is that tests in the TESTING package should be modifiable by the maintainer of the TESTED package, with both packages suspended if the two maintainers cannot agree. We need to be able to move forward when legacy behaviour is outdated or just plain wrong. Or, in the case that I find affects me, when improvements in iterative schemes change iterates slightly. My guess is that Duncan's example is a case in point. I doubt this will ever occur, as it doesn't seem to be the R way. However, I do know that improvements in methods are not going to CRAN in some cases. JN __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
An offline message said "Oh, you mean you're asking for an actual object-oriented language that hides the internal representations of objects from users of a package. Unfortunately for all of us, that isn't R." Actually, I don't think that's what I want. We already have lots of languages like that (and it's not too hard to integrate C++ with R, for example), but I'm not tempted to move code from R to C++ unnecessarily: there's too much mental overhead (for me) in handling the extra formality. What I think I'd like is something closer to the :: versus ::: difference in R. In practice, it's purely a convention that ::: isn't used. The package writer declares (via the NAMESPACE) that certain functions are fair game and will tend to be stable over time, but can't block access to the internal functions that aren't: users are free to take chances and use :::. I'd like to be able to declare that the order of components in a list doesn't matter. I think the NULL component versus missing component is basically a bug in the language, but it's such an old one that I don't think it could be changed: so I'd like to be able to declare that it's not a distinction anyone should depend on. Both of these things could easily be handled by the existing S3 machinery. Instead of declaring my object to have class "mesh3d", I could declare it to have class c("mesh3d", "unorderedList", "ignoreNULL"), and comparison methods like base::all.equal() or waldo::compare() would ignore the unimportant differences. The hardest thing here would be to agree on the class names and what they imply. Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
> On 16 Apr 2021, at 14:06, Duncan Murdoch wrote: > > On 16/04/2021 7:54 a.m., Hugh Parsonage wrote: >> My 2c is that if a package has actually tested something like that >> (say, the ordering of a list's elements), then it is 100% likely that >> a script out there depends on that behaviour too. > > I suspect this test is not intentional, and it really wouldn't make sense for > a package to depend on the order of named list components. You could randomise the order…. > > In other words, the >> change is not inconsequential. R users are in debt to package >> developers but I think it behoves us to take special care not to >> underestimate the work involved when a package's behaviour changes. > > There's a difference between internal implementation changes and external > behaviour changes. For example, there are clear warnings not to use > un-exported functions (via :::), and a package that did that would not be > accepted on CRAN. > > What I'm asking for are suggestions for ways to hide internal layouts of > objects from users. That seems a lot harder. > > Duncan Murdoch > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
On 16/04/2021 7:54 a.m., Hugh Parsonage wrote: My 2c is that if a package has actually tested something like that (say, the ordering of a list's elements), then it is 100% likely that a script out there depends on that behaviour too. I suspect this test is not intentional, and it really wouldn't make sense for a package to depend on the order of named list components. In other words, the change is not inconsequential. R users are in debt to package developers but I think it behoves us to take special care not to underestimate the work involved when a package's behaviour changes. There's a difference between internal implementation changes and external behaviour changes. For example, there are clear warnings not to use un-exported functions (via :::), and a package that did that would not be accepted on CRAN. What I'm asking for are suggestions for ways to hide internal layouts of objects from users. That seems a lot harder. Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
On 16/04/2021 7:40 a.m., brodie gaslam wrote: An all.equal method? This might not work with 3rd edition though (untested), and I'm not sure what method registration requirements would exist for the user. It already has one, but it isn't used by testthat 3. Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
My 2c is that if a package has actually tested something like that (say, the ordering of a list's elements), then it is 100% likely that a script out there depends on that behaviour too. In other words, the change is not inconsequential. R users are in debt to package developers but I think it behoves us to take special care not to underestimate the work involved when a package's behaviour changes. On Fri, 16 Apr 2021 at 20:09, Duncan Murdoch wrote: > > I'm updating the rgl package, and have come across the following problem. > > Some packages that depend on rgl and do careful testing are detecting > inconsequential changes. A typical case is the nat package, which has > extensive testing, some of which comes down to checking results from rgl > functions using testthat::is_equal(). I rewrote some parts of the rgl > code, and so some rgl objects differ in irrelevant ways. For example, > >obj <- list(x = NULL) > > differs from > >obj <- list() >obj[["x"]] <- NULL > > (the first is length 1, the second ends up with no entries). All tests > in rgl would be like > >if (is.null(obj[["x"]])) ... > > so the tests evaluate the same, but testthat::is_equal() doesn't return > TRUE, because the objects are different. > > Another example would be > >obj <- list() >obj$a <- 1 >obj$b <- 2 > > which differs from > >obj <- list() >obj$b <- 2 >obj$a <- 1 > > in the order of the components, but since rgl always accessed them by > name, that makes no difference. > > So what I'm looking for is a strategy to hide internal implementation > details. (I'm using a compare_proxy() method now which standardizes the > objects, but that seems very specific to testthat version 3. I'd like > ideas for something more robust.) > > Duncan Murdoch > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Problems with too much testing
An all.equal method? This might not work with 3rd edition though (untested), and I'm not sure what method registration requirements would exist for the user. Best, B. On Friday, April 16, 2021, 6:09:24 AM EDT, Duncan Murdoch wrote: I'm updating the rgl package, and have come across the following problem. Some packages that depend on rgl and do careful testing are detecting inconsequential changes. A typical case is the nat package, which has extensive testing, some of which comes down to checking results from rgl functions using testthat::is_equal(). I rewrote some parts of the rgl code, and so some rgl objects differ in irrelevant ways. For example, obj <- list(x = NULL) differs from obj <- list() obj[["x"]] <- NULL (the first is length 1, the second ends up with no entries). All tests in rgl would be like if (is.null(obj[["x"]])) ... so the tests evaluate the same, but testthat::is_equal() doesn't return TRUE, because the objects are different. Another example would be obj <- list() obj$a <- 1 obj$b <- 2 which differs from obj <- list() obj$b <- 2 obj$a <- 1 in the order of the components, but since rgl always accessed them by name, that makes no difference. So what I'm looking for is a strategy to hide internal implementation details. (I'm using a compare_proxy() method now which standardizes the objects, but that seems very specific to testthat version 3. I'd like ideas for something more robust.) Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] Problems with too much testing
I'm updating the rgl package, and have come across the following problem. Some packages that depend on rgl and do careful testing are detecting inconsequential changes. A typical case is the nat package, which has extensive testing, some of which comes down to checking results from rgl functions using testthat::is_equal(). I rewrote some parts of the rgl code, and so some rgl objects differ in irrelevant ways. For example, obj <- list(x = NULL) differs from obj <- list() obj[["x"]] <- NULL (the first is length 1, the second ends up with no entries). All tests in rgl would be like if (is.null(obj[["x"]])) ... so the tests evaluate the same, but testthat::is_equal() doesn't return TRUE, because the objects are different. Another example would be obj <- list() obj$a <- 1 obj$b <- 2 which differs from obj <- list() obj$b <- 2 obj$a <- 1 in the order of the components, but since rgl always accessed them by name, that makes no difference. So what I'm looking for is a strategy to hide internal implementation details. (I'm using a compare_proxy() method now which standardizes the objects, but that seems very specific to testthat version 3. I'd like ideas for something more robust.) Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel