Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20410 I think that the problem is not SPARK-22797. The problem is that before this PR, the Python API considers as Defined but not Set all the parameters with a default value, while the Scala/Java class representing it considers as Set all them. This has come up in this case, but it can cause other problems in the future and also now, because it creates an inconsistency between the Python API and the representation in the JVM backend. Thus I do believe that this PR is needed, and it is not only a fix for the test failures. I think this is a first step and a second step would later be to drop all the `setDefault` in the Python API, in favor of retrieving them from the JVM backend. In this way, we will be sure there is no logical inconsistency between the API and the backend. Unfortunately, this second part is much bigger and has a large impact. So I think it best would need a design doc or something similar.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org