davidm-db commented on code in PR #54905:
URL: https://github.com/apache/spark/pull/54905#discussion_r3136338496
##########
sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala:
##########
@@ -123,4 +163,18 @@ object TypeApiOps {
case _ => None
}
}
+
+ /**
+ * Reverse lookup: converts an Arrow type to a Spark DataType.
+ */
+ def fromArrowType(at: ArrowType): Option[DataType] = {
+ import org.apache.arrow.vector.types.TimeUnit
+ if (!SqlApiConf.get.typesFrameworkEnabled) return None
Review Comment:
I will do that. I will create a follow-up PR as soon as this one is merged,
to address this issue in all relevant framework traits.
As we agreed offline, I will do the approach with `lazy val` since it's
least invasive to mitigate the perf issue immediately. And afterwards, I'll
follow up with a more thorough benchmarking to check if the `supports` approach
is more efficient. For now, there's no evidence based on the benchmarks I ran,
but I'll confirm better.
That's why I think a separate PR would be nice, to document this topic in
the PR description nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
