ryan-johnson-databricks commented on code in PR #43334:
URL: https://github.com/apache/spark/pull/43334#discussion_r1447667682
##########
sql/core/src/main/scala/org/apache/spark/sql/package.scala:
##########
@@ -73,4 +76,41 @@ package object sql {
* with rebasing.
*/
private[sql] val SPARK_LEGACY_INT96_METADATA_KEY =
"org.apache.spark.legacyINT96"
+
+ /**
+ * This helper function captures the Spark API and its call site in the user
code from the current
+ * stacktrace.
+ *
+ * As adding `withOrigin` explicitly to all Spark API definition would be a
huge change,
+ * `withOrigin` is used only at certain places where all API implementation
surely pass through
+ * and the current stacktrace is filtered to the point where first Spark API
code is invoked from
+ * the user code.
+ *
+ * As there might be multiple nested `withOrigin` calls (e.g. any Spark API
implementations can
+ * invoke other APIs) only the first `withOrigin` is captured because that
is closer to the user
+ * code.
+ *
+ * @param f The function that can use the origin.
+ * @return The result of `f`.
+ */
+ private[sql] def withOrigin[T](f: => T): T = {
+ if (CurrentOrigin.get.stackTrace.isDefined) {
+ f
+ } else {
+ val st = Thread.currentThread().getStackTrace
+ var i = 3
+ while (i < st.length && sparkCode(st(i))) i += 1
+ val origin =
+ Origin(stackTrace = Some(Thread.currentThread().getStackTrace.slice(i
- 1, i + 1)))
Review Comment:
Isn't this super expensive, calling `currentThread().getStackTrace` in a
loop?? Can't we grab the stacktrace only once, and filter it as needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
