Yay! That's right!!! Thanks Reynold. Such a short answer with so much information. Thanks.
Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski On Wed, May 30, 2018 at 8:10 PM, Reynold Xin <r...@databricks.com> wrote: > SQL expressions? > > On Wed, May 30, 2018 at 11:09 AM Jacek Laskowski <ja...@japila.pl> wrote: > >> Hi, >> >> I've been exploring RuntimeReplaceable expressions [1] and have been >> wondering what their purpose is. >> >> Quoting the scaladoc [2]: >> >> > An expression that gets replaced at runtime (currently by the >> optimizer) into a different expression for evaluation. This is mainly used >> to provide compatibility with other databases. >> >> For example, ParseToTimestamp expression is a RuntimeReplaceable >> expression and it is replaced by Cast(left, TimestampType) >> or Cast(UnixTimestamp(left, format), TimestampType) per to_timestamp >> function (there are two variants). >> >> My question is why is this RuntimeReplaceable better than simply using >> the Casts as the implementation of to_timestamp functions? >> >> def to_timestamp(s: Column, fmt: String): Column = withExpr { >> // pseudocode >> Cast(UnixTimestamp(left, format), TimestampType) >> } >> >> What's wrong with the above implementation compared to the current one? >> >> [1] https://github.com/apache/spark/blob/master/sql/ >> catalyst/src/main/scala/org/apache/spark/sql/catalyst/ >> expressions/Expression.scala#L275 >> >> [2] https://github.com/apache/spark/blob/master/sql/ >> catalyst/src/main/scala/org/apache/spark/sql/catalyst/ >> expressions/Expression.scala#L266-L267 >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://about.me/JacekLaskowski >> Mastering Spark SQL https://bit.ly/mastering-spark-sql >> Spark Structured Streaming https://bit.ly/spark-structured-streaming >> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams >> Follow me at https://twitter.com/jaceklaskowski >> >
