Spenserrrr commented on code in PR #58900:
URL: https://github.com/apache/spark/pull/58900#discussion_r4051294421
##########
python/pyspark/eval_handlers/__init__.py:
##########
@@ -22,7 +22,18 @@
Each eval type handled here is an ``EvalTypeHandler`` subclass (in ``_base``)
that
declares its ``eval_type`` and self-registers at class definition, which
``read_udfs`` looks up via ``get_eval_type_handler``. Importing this package
-imports the concrete handler submodules (``_arrow``) so they register.
+imports the concrete handler submodules so they register.
+
+``_arrow`` requires pyarrow and imports it at module top, so it is only
imported
+when pyarrow is available; the Arrow eval types it serves cannot run without
it.
"""
-from pyspark.eval_handlers import _arrow # noqa: F401 # registers handlers
on import
+try:
+ from pyspark.sql.pandas.utils import require_minimum_pyarrow_version
+
+ require_minimum_pyarrow_version()
+except Exception:
Review Comment:
Could we decouple handler registration from PyArrow availability? This catch
silently leaves the built-in Arrow handlers unregistered, so a valid Arrow eval
type can later fall through to `ValueError("Unknown eval type")` instead of
reporting the missing or incompatible PyArrow dependency. Could `_arrow` remain
importable for registration and defer the PyArrow check until a handler is
instantiated or run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
