Hi, Just noticed that http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html (Spark 2.4.5) has formatting issues in "Upgrading from Spark SQL 2.4.3 to 2.4.4" [1] which got fixed in master [2]. That's OK.
What made me wonder was the other change to the section "Upgrading from Spark SQL 2.4 to 2.4.5" [3] that had the following item included: "Starting from 2.4.5, SQL configurations are effective also when a Dataset is converted to an RDD and its plan is executed due to action on the derived RDD. The previous behavior can be restored setting spark.sql.legacy.rdd.applyConf to false: in this case, SQL configurations are ignored for operations performed on a RDD derived from a Dataset." Why was this removed in master [4]? It was mentioned in "Notable changes" of Spark Release 2.4.5 [5]. [1] http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html#upgrading-from-spark-sql-243-to-244 [2] https://github.com/apache/spark/blob/master/docs/sql-migration-guide.md#upgrading-from-spark-sql-243-to-244 [3] http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html#upgrading-from-spark-sql-24-to-245 [4] https://github.com/apache/spark/blob/master/docs/sql-migration-guide.md#upgrading-from-spark-sql-244-to-245 [5] http://spark.apache.org/releases/spark-release-2-4-5.html Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
