Hyukjin Kwon created SPARK-27834:
------------------------------------
Summary: Make separate PySpark/SparkR vectorization configurations
Key: SPARK-27834
URL: https://issues.apache.org/jira/browse/SPARK-27834
Project: Spark
Issue Type: Improvement
Components: PySpark, SparkR, SQL
Affects Versions: 3.0.0
Reporter: Hyukjin Kwon
{{spark.sql.execution.arrow.enabled}} was added when we add PySpark arrow
optimization.
Later, in the current master, SparkR arrow optimization was added and it's
controlled by the same configuration {{spark.sql.execution.arrow.enabled}}.
There look two issues about this:
1. {{spark.sql.execution.arrow.enabled}} in PySpark was added from 2.3.0
whereas SparkR optimization was added 3.0.0. The stability is different so it's
problematic when we change the default value for one of both optimization first.
2. Suppose users want to share some JVM by PySpark and SparkR. They are
currently forced to use the optimization for all or none.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
