cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1565491650
########## sql/core/src/main/scala/org/apache/spark/sql/Column.scala: ########## @@ -171,6 +171,29 @@ class Column(val expr: Expression) extends Logging { Column.fn(name, this, lit(other)) } + /** + * A version of the `fn` method specifically designed for binary operations in PySpark + * that require logging information. + * This method is used when the operation involves another Column. + * + * @param name The name of the operation to be performed. + * @param other The value to be used in the operation, which will be converted to a + * Column if not already one. + * @param pysparkFragment A string representing the 'fragment' of the PySpark error context, + * typically indicates the name of PySpark function. + * @param pysparkCallSite A string representing the 'callSite' of the PySpark error context, + * providing the exact location within the PySpark code where the + * operation originated. + * @return A Column resulting from the operation. + */ + private def fn( Review Comment: @HyukjinKwon This probably can't cover all the cases, and we may need to add more overloads for certain functions that require non-expression parameters, but it shouldn't be any. I think it's better than using ThreadLocal which can be quite fragile to pass values between Python and JVM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org