It does make sense, I would go a little further and make this
field/property a single value of the same type than the array. This
would allow using any arbitrary sentinel value for unknown values (0
in your suggested case). The end result is zero-copy for R bindings
(if stars are aligned). I created ARROW-8348 [1] for this.
François
[1] https://jira.apache.org/jira/browse/ARROW-8348
On Mon, Apr 6, 2020 at 11:02 AM Felix Benning <felix.benn...@gmail.com>
wrote:
Would it make sense to have an `na_are_zero` flag? Since null checking is
not without cost, it might be helpful to some algorithms, if the content
"underneath" the nulls is zero. For example in means, or scalar products
and thus matrix multiplication, knowing that the array has zeros where
the
na's are, would allow these algorithms to pretend that there are no na's.
Since setting all nulls to zero in a matrix of n columns and n rows costs
O(n^2), it would make sense to set them all to zero before matrix
multiplication i.e. O(n^3) and similarly expensive algorithms. If there
was
a `na_are_zero` flag, other algorithms could later utilize this work
already being done. Algorithms which change the data and violate this
contract, would only need to reset the flag. And in some use cases, it
might be possible to use idle time of the computer to "clean up" the
na's,
preparing for the next query.
Felix
---------- Forwarded message ---------
From: Wes McKinney <wesmck...@gmail.com>
Date: Sun, 5 Apr 2020 at 22:31
Subject: Re: Attn: Wes, Re: Masked Arrays
To: <u...@arrow.apache.org>
As I recall the contents "underneath" have been discussed before and
the consensus was that the contents are not specified. If you'e like
to make a proposal to change something I would suggest raising it on
dev@arrow.apache.org
On Sun, Apr 5, 2020 at 1:56 PM Felix Benning <felix.benn...@gmail.com>
wrote:
Follow up: Do you think it would make sense to have an `na_are_zero`
flag? Since it appears that the baseline (naively assuming there are no
null values) is still a bit faster than equally optimized null value
handling algorithms. So you might want to make the assumption, that all
null values are set to zero in the array (instead of undefined). This
would
allow for very fast means, scalar products and thus matrix multiplication
which ignore nas. And in case of matrix multiplication, you might prefer
sacrificing an O(n^2) effort to set all null entries to zero before
multiplying. And assuming you do not overwrite this data, you would be
able
to reuse that assumption in later computations with such a flag.
In some use cases, you might even be able to utilize unused computing
resources for this task. I.e. clean up the nulls while the computer is
not
used, preparing for the next query.
On Sun, 5 Apr 2020 at 18:34, Felix Benning <felix.benn...@gmail.com>
wrote:
Awesome, that was exactly what I was looking for, thank you!
On Sun, 5 Apr 2020 at 00:40, Wes McKinney <wesmck...@gmail.com>
wrote:
I wrote a blog post a couple of years about this
https://wesmckinney.com/blog/bitmaps-vs-sentinel-values/
Pasha Stetsenko did a follow-up analysis that showed that my
"sentinel" code could be significantly improved, see:
https://github.com/st-pasha/microbench-nas/blob/master/README.md
Generally speaking in Apache Arrow we've been happy to have a uniform
representation of nullness across all types, both primitive
(booleans,
numbers, or strings) and nested (lists, structs, unions, etc.). Many
computational operations (like elementwise functions) need not
concern
themselves with the nulls at all, for example, since the bitmap from
the input array can be passed along (with zero copy even) to the
output array.
On Sat, Apr 4, 2020 at 4:39 PM Felix Benning <
felix.benn...@gmail.com>
wrote:
Does anyone have an opinion (or links) about Bitpattern vs Masked
Arrays for NA implementations? There seems to have been a discussion
about
that in the numpy community in 2012
https://numpy.org/neps/nep-0026-missing-data-summary.html without an
apparent result.
Summary of the Summary:
- The Bitpattern approach reserves one bitpattern of any type as
na,
the only type not having spare bitpatterns are integers which means this
decreases their range by one. This approach is taken by R and was
regarded
as more performant in 2012.
- The Mask approach was deemed more flexible, since it would allow
"degrees of missingness", and also cleaner/easier implementation.
Since bitpattern checks would probably disrupt SIMD, I feel like
some
calculations (e.g. mean) would actually benefit more, from setting na
values to zero, proceeding as if they were not there, and using the
number
of nas in the metadata to adjust the result. This of course does not work
if two columns are used (e.g. scalar product), which is probably more
important.
Was using Bitmasks in Arrow a conscious performance decision? Or
was
the decision only based on the fact, that R and Bitpattern
implementations
in general are a niche, which means that Bitmasks are more compatible
with
other languages?
I am curious about this topic, since the "lack of proper na
support"
was cited as the reason, why Python would never replace R in statistics.
Thanks,
Felix
On 31.03.20 14:52, Joris Van den Bossche wrote:
Note that pandas is starting to use a notion of "masked arrays" as
well, for example for its nullable integer data type, but also not using
the np.ma masked array, but a custom implementation (for technical
reasons
in pandas this was easier).
Also, there has been quite some discussion last year in numpy
about a
possible re-implementation of a MaskedArray, but using numpy's protocols
(`__array_ufunc__`, `__array_function__` etc), instead of being a
subclass
like np.ma now is. See eg
https://mail.python.org/pipermail/numpy-discussion/2019-June/079681.html
.
Joris
On Mon, 30 Mar 2020 at 18:57, Daniel Nugent <nug...@gmail.com>
wrote:
Ok. That actually aligns closely to what I'm familiar with. Good
to
know.
Thanks again for taking the time to respond,
-Dan Nugent
On Mon, Mar 30, 2020 at 12:38 PM Wes McKinney <
wesmck...@gmail.com>
wrote:
Social and technical reasons I guess. Empirically it's just not
used much.
You can see my comments about numpy.ma in my 2010 paper about
pandas
https://conference.scipy.org/proceedings/scipy2010/pdfs/mckinney.pdf
At least in 2010, there were notable performance problems when
using
MaskedArray for computations
"We chose to use NaN as opposed to using NumPy MaskedArrays for
performance reasons (which are beyond the scope of this paper),
as
NaN
propagates in floating-point operations in a natural way and can
be
easily detected in algorithms."
On Mon, Mar 30, 2020 at 11:20 AM Daniel Nugent <nug...@gmail.com
wrote:
Thanks! Since I'm just using it to jump to Arrow, I think I'll
stick with it.
Do you have any feelings about why Numpy's masked arrays didn't
gain favor when many data representation formats explicitly support
nullity
(including Arrow)? Is it just that not carrying nulls in computations
forward is preferable (that is, early filtering/value filling was
easier)?
-Dan Nugent
On Mon, Mar 30, 2020 at 11:40 AM Wes McKinney <
wesmck...@gmail.com>
wrote:
On Mon, Mar 30, 2020 at 8:31 AM Daniel Nugent <
nug...@gmail.com>
wrote:
Didn’t want to follow up on this on the Jira issue earlier
since it's sort of tangential to that bug and more of a usage question.
You
said:
I wouldn't recommend building applications based on them
nowadays since the level of support / compatibility in other projects is
low.
In my case, I am using them since it seemed like a
straightforward representation of my data that has nulls, the format I’m
converting from has zero cost numpy representations, and converting from
an
internal format into Arrow in memory structures appears zero cost (or
close
to it) as well. I guess I can just provide the mask as an explicit
argument, but my original desire to use it came from being able to
exploit
numpy.ma.concatenate in a way that saved some complexity in
implementation.
Since Arrow itself supports masking values with a bitfield,
is
there something intrinsic to the notion of array masks that is not well
supported? Or do you just mean the specific numpy MaskedArray class?
I mean just the numpy.ma module. Not many Python computing
projects
nowadays treat MaskedArray objects as first class citizens.
Depending
on what you need it may or may not be a problem. pyarrow
supports
ingesting from MaskedArray as a convenience, but it would not
be
common in my experience for a library's APIs to return
MaskedArrays.
If this is too much of a numpy question rather than an arrow
question, could you point me to where I can read up on masked array
support
or maybe what the right place to ask the numpy community about whether
what
I'm doing is appropriate or not.
Thanks,
-Dan Nugent
