[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586091#comment-16586091 ]
Apache Spark commented on SPARK-25140: -------------------------------------- User 'maropu' has created a pull request for this issue: https://github.com/apache/spark/pull/22154 > Add optional logging to UnsafeProjection.create when it falls back to > interpreted mode > -------------------------------------------------------------------------------------- > > Key: SPARK-25140 > URL: https://issues.apache.org/jira/browse/SPARK-25140 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.0 > Reporter: Kris Mok > Priority: Minor > > SPARK-23711 implemented a nice graceful handling of allowing UnsafeProjection > to fall back to an interpreter mode when codegen fails. That makes Spark much > more usable even when codegen is unable to handle the given query. > But in its current form, the fallback handling can also be a mystery in terms > of performance cliffs. Users may be left wondering why a query runs fine with > some expressions, but then with just one extra expression the performance > goes 2x, 3x (or more) slower. > It'd be nice to have optional logging of the fallback behavior, so that for > users that care about monitoring performance cliffs, they can opt-in to log > when a fallback to interpreter mode was taken. i.e. at > https://github.com/apache/spark/blob/a40ffc656d62372da85e0fa932b67207839e7fde/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L183 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org