Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20839 ### After Reworded Warnings ``` In [2]: spark.createDataFrame(pd.DataFrame([[{u'a': 1}]]), "a: map<string, int>") /home/bryan/git/spark/python/pyspark/sql/session.py:688: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true, but has reached the error below and will not continue because automatic fallback with 'spark.sql.execution.arrow.fallback.enabled' has been set to false. PyArrow >= 0.8.0 must be installed; however, it was not found. warnings.warn(msg) --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-2-ecc28a9b5e18> in <module>() ----> 1 spark.createDataFrame(pd.DataFrame([[{u'a': 1}]]), "a: map<string, int>") ~/git/spark/python/pyspark/sql/session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema) 665 and len(data) > 0: 666 try: --> 667 return self._create_from_pandas_with_arrow(data, schema, timezone) 668 except Exception as e: 669 from pyspark.util import _exception_message ~/git/spark/python/pyspark/sql/session.py in _create_from_pandas_with_arrow(self, pdf, schema, timezone) 508 509 require_minimum_pandas_version() --> 510 require_minimum_pyarrow_version() 511 512 from pandas.api.types import is_datetime64_dtype, is_datetime64tz_dtype ~/git/spark/python/pyspark/sql/utils.py in require_minimum_pyarrow_version() 147 if not have_arrow: 148 raise ImportError("PyArrow >= %s must be installed; however, " --> 149 "it was not found." % minimum_pyarrow_version) 150 if LooseVersion(pyarrow.__version__) < LooseVersion(minimum_pyarrow_version): 151 raise ImportError("PyArrow >= %s must be installed; however, " ImportError: PyArrow >= 0.8.0 must be installed; however, it was not found. ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org