Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21427
> But we marked this as experimental ...
That's also special for this case, we marked it as experimental in 2.3.1.
Not a lot of behavior changes are similar to this one. To highlight:
1. it's not marked as experimental in the first release.
2. it missed 2.3.1, so the old behavior will be there for some time, until
the next release(2.3.2 or 2.4.0)
3. it turns runnable code into failure, and the old behavior is kind of
self-consistent(by-position match). it's not like turning failures into
runnable or fix a correctness bug.
To summary:
1. I agree the new behavior makes more sense, we should have done that in
the first place.
2. This is a special case like I mentioned above. We should be a little
more conservative here.
3. Adding a config is not hard. Maybe @ueshin can build the framework first
for passing configs to the python worker?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
