Spenserrrr commented on PR #58485: URL: https://github.com/apache/spark/pull/58485#issuecomment-5515944868
Hi @zhengruifeng! This PR adds a per-ufunc table of the operand types NumPy accepts and checks it in the dispatch, so an unsupported type raises TypeError instead of being cast and computed. There are three things I want to point out: - The two commits are split by release status, and the second one has a behavior change in the released version. It can be dropped if you would rather ship only the first one. - One alternative was putting the check inline in each mapping function, as a leading when(~typeof(c).isin(...), raise_error(...)). That rejects the same types, but it raises lazily: `np.fmod(str_col, str_col)` returns a Series, and the error only appears at `to_pandas()` as a SparkRuntimeException. In contrast, pandas raises TypeError at the call itself. Also, a mapping function receives Columns, so it cannot raise in Python, while the dispatch still has the Series and its data type. Happy to switch to the inline form if you prefer it. - The boolean gap noted in the description is the follow-up I plan to file; the operand cast that fixes it was reverted earlier for letting timestamps through, which this check prevents. Please let me know if you don't want this change. Could you take a look when you have time? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
