we were running in to an NPE in one of our UDFs for spark sql. now this particular function indeed could not handle nulls, but this was by design since null input was never allowed (and we would want it to blow up if there was a null as input).
we realized the issue was not in our data when we added filters for nulls and the NPE still happened. then we also saw the NPE when just doing dataframe.explain instead of running our job. turns out the issue is in EliminateOuterJoin.canFilterOutNull where a row with all nulls ifs fed into the expression as a test. its the line: val v = boundE.eval(emptyRow) so should we conclude from this that all udfs should always be prepared to handle nulls?
