This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2e5825fb32c0 [SPARK-47858][PYTHON][FOLLOWUP] Excluding Python magic
methods from error context target
2e5825fb32c0 is described below
commit 2e5825fb32c03faa854921c54feb6ad0e7b5d432
Author: Haejoon Lee <haejoon....@databricks.com>
AuthorDate: Fri Apr 26 16:53:22 2024 +0900
[SPARK-47858][PYTHON][FOLLOWUP] Excluding Python magic methods from error
context target
### What changes were proposed in this pull request?
This PR followups for https://github.com/apache/spark/pull/46063 to exclude
Python magic methods from error context target
### Why are the changes needed?
We only need to logging the Apache Spark Column API functions for
DataFrameQueryContext, but not Python native functions.
### Does this PR introduce _any_ user-facing change?
No API changes, but the error message only contain Column APIs that are
supported by Apache Spark
### How was this patch tested?
The existing CI should pass.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46215 from itholic/47858-followup.
Authored-by: Haejoon Lee <haejoon....@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/errors/utils.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/errors/utils.py b/python/pyspark/errors/utils.py
index 08744dc4c264..180bdd681306 100644
--- a/python/pyspark/errors/utils.py
+++ b/python/pyspark/errors/utils.py
@@ -197,6 +197,14 @@ def with_origin_to_class(cls: Type[T]) -> Type[T]:
"""
if os.environ.get("PYSPARK_PIN_THREAD", "true").lower() == "true":
for name, method in cls.__dict__.items():
- if callable(method) and name != "__init__":
+ # Excluding Python magic methods that do not utilize JVM functions.
+ if callable(method) and name not in (
+ "__init__",
+ "__new__",
+ "__iter__",
+ "__nonzero__",
+ "__repr__",
+ "__bool__",
+ ):
setattr(cls, name, _with_origin(method))
return cls
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
