Dear Adrian, SPSS and other packages handle this problem in a very similar way to what I described: they store additional metadata for each variable. You can see this in the way that SPSS organizes it's file format: each "variable" has additional metadata that indicate how specific values of the variable, encoded as an integer or a floating point should be handled in analysis. Before you actually run a crosstab in SPSS, the metadata is (presumably) applied to the raw data to arrive at an in memory buffer on which the actual model is fitted, etc.
The 20 line solution in R looks like this: df <- data.frame(q1 = c(1, 10, 50, 999), q2 = c("Yes", "No", "Don't know", "Interviewer napping"), stringsAsFactors = FALSE) attr(df$q1, 'missing') <- 999 attr(df$q2, 'missing') <- c("Don't know", "Interviewer napping") excludeMissing <- function(df) { for(q in names(df)) { v <- df[[q]] mv <- attr(v, 'missing') if(!is.null(mv)) { df[[q]] <- ifelse(v %in% mv, NA, v) } } df } table(excludeMissing(df)) If you want to preserve the missing attribute when subsetting the vectors then you will have to take the example further by adding a class and `[.withMissing` functions. This might bring the whole project to a few hundred lines, but the rules that apply here are well defined and well understood, giving you a proper basis on which to build. And perhaps the vctrs package might make this even simpler, take a look. Best, Alex On Mon, May 24, 2021 at 3:20 PM Taras Zakharko <taras.zakha...@uzh.ch> wrote: > Hi Adrian, > > Have a look at vctrs package — they have low-level primitives that might > simplify your life a bit. I think you can get quite far by creating a > custom type that stores NAs in an attribute and utilizes vctrs proxy > functionality to preserve these attributes across different operations. > Going that route will likely to give you a much more flexible and robust > solution. > > Best, > > Taras > > > On 24 May 2021, at 15:09, Adrian Dușa <dusa.adr...@gmail.com> wrote: > > > > Dear Alex, > > > > Thanks for piping in, I am learning with each new message. > > The problem is clear, the solution escapes me though. I've already tried > > the attributes route: it is going to triple the data size: along with the > > additional (logical) variable that specifies which level is missing, one > > also needs to store an index such that sorting the data would still > > maintain the correct information. > > > > One also needs to think about subsetting (subset the attributes as well), > > splitting (the same), aggregating multiple datasets (even more > attention), > > creating custom vectors out of multiple variables... complexity quickly > > grows towards infinity. > > > > R factors are nice indeed, but: > > - there are numerical variables which can hold multiple missing values > (for > > instance income) > > - factors convert the original questionnaire values: if a missing value > was > > coded 999, turning that into a factor would convert that value into > > something else > > > > I really, and wholeheartedly, do appreciate all advice: but please be > > assured that I have been thinking about this for more than 10 years and > > still haven't found a satisfactory solution. > > > > Which makes it even more intriguing, since other software like SAS or > Stata > > have solved this for decades: what is their implementation, and how come > > they don't seem to be affected by the new M1 architecture? > > When package "haven" introduced the tagged NA values I said: ah-haa... so > > that is how it's done... only to learn that implementation is just as > > fragile as the R internals. > > > > There really should be a robust solution for this seemingly mundane > > problem, but apparently is far from mundane... > > > > Best wishes, > > Adrian > > > > > > On Mon, May 24, 2021 at 3:29 PM Bertram, Alexander < > a...@bedatadriven.com> > > wrote: > > > >> Dear Adrian, > >> I just wanted to pipe in and underscore Thomas' point: the payload bits > of > >> IEEE 754 floating point values are no place to store data that you care > >> about or need to keep. That is not only related to the R APIs, but also > how > >> processors handle floating point values and signaling and non-signaling > >> NaNs. It is very difficult to reason about when and under which > >> circumstances these bits are preserved. I spent a lot of time working on > >> Renjin's handling of these values and I can assure that any such scheme > >> will end in tears. > >> > >> A far, far better option is to use R's attributes to store this kind of > >> metadata. This is exactly what this language feature is for. There is > >> already a standard 'levels' attribute that holds the labels of factors > like > >> "Yes", "No" , "Refused", "Interviewer error'' etc. In the past, I've > worked > >> on projects where we stored an additional attribute like "missingLevels" > >> that stores extra metadata on which levels should be used in which kind > of > >> analysis. That way, you can preserve all the information, and then > write a > >> utility function which automatically applies certain logic to a whole > >> dataframe just before passing the data to an analysis function. This is > >> also important because in surveys like this, different values should be > >> excluded at different times. For example, you might want to include all > >> responses in a data quality report, but exclude interviewer error and > >> refusals when conducting a PCA or fitting a model. > >> > >> Best, > >> Alex > >> > >> On Mon, May 24, 2021 at 2:03 PM Adrian Dușa <dusa.adr...@gmail.com> > wrote: > >> > >>> On Mon, May 24, 2021 at 1:31 PM Tomas Kalibera < > tomas.kalib...@gmail.com> > >>> wrote: > >>> > >>>> [...] > >>>> > >>>> For the reasons I explained, I would be against such a change. Keeping > >>> the > >>>> data on the side, as also recommended by others on this list, would > >>> allow > >>>> you for a reliable implementation. I don't want to support fragile > >>> package > >>>> code building on unspecified R internals, and in this case > particularly > >>>> internals that themselves have not stood the test of time, so are at > >>> high > >>>> risk of change. > >>>> > >>> I understand, and it makes sense. > >>> We'll have to wait for the R internals to settle (this really is > >>> surprising, I wonder how other software have solved this). In the > >>> meantime, > >>> I will probably go ahead with NaNs. > >>> > >>> Thank you again, > >>> Adrian > >>> > >>> [[alternative HTML version deleted]] > >>> > >>> ______________________________________________ > >>> R-devel@r-project.org mailing list > >>> https://stat.ethz.ch/mailman/listinfo/r-devel > >>> > >> > >> > >> -- > >> Alexander Bertram > >> Technical Director > >> *BeDataDriven BV* > >> > >> Web: http://bedatadriven.com > >> Email: a...@bedatadriven.com > >> Tel. Nederlands: +31(0)647205388 > >> Skype: akbertram > >> > > > > [[alternative HTML version deleted]] > > > > ______________________________________________ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > ______________________________________________ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > -- Alexander Bertram Technical Director *BeDataDriven BV* Web: http://bedatadriven.com Email: a...@bedatadriven.com Tel. Nederlands: +31(0)647205388 Skype: akbertram [[alternative HTML version deleted]] ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel