HyukjinKwon commented on code in PR #57800:
URL: https://github.com/apache/spark/pull/57800#discussion_r3754694217
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -5390,6 +5390,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val PYTHON_UDF_MAP_IN_BATCH_LEGACY_ACCEPT_ANY_ITERABLE_ENABLED =
+
buildConf("spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled")
+ .internal()
+ .doc("When true, mapInPandas and mapInArrow UDFs may return any iterable
(e.g. a list) " +
+ "rather than a strict iterator, matching the behavior before 4.3.0.
When false, the " +
+ "returned value must be an iterator, matching the declared
Iterator[...] signatures.")
+ .version("4.3.0")
Review Comment:
ah ok. let's land this to 4.3. synced w @Yicong-Huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
