[
https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kris Mok updated SPARK-25140:
-----------------------------
Description:
SPARK-23711 implemented a nice graceful handling of allowing UnsafeProjection
to fall back to an interpreter mode when codegen fails. That makes Spark much
more usable even when codegen is unable to handle the given query.
But in its current form, the fallback handling can also be a mystery in terms
of performance cliffs. Users may be left wondering why a query runs fine with
some expressions, but then with just one extra expression the performance goes
2x, 3x (or more) slower.
It'd be nice to have optional logging of the fallback behavior, so that for
users that care about monitoring performance cliffs, they can opt-in to log
when a fallback to interpreter mode was taken. i.e. at
https://github.com/apache/spark/blob/a40ffc656d62372da85e0fa932b67207839e7fde/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L183
was:
SPARK-23711 implemented a nice graceful handling of allowing UnsafeProjection
to fall back to an interpreter mode when codegen fails. That makes Spark much
more usable even when codegen is unable to handle the given query.
But in its current form, the fallback handling can also be a mystery in terms
of performance cliffs. Users may be left wondering why a query runs fine with
some expressions, but then with just one extra expression the performance goes
2x, 3x (or more) slower.
It'd be nice to have optional logging of the fallback behavior, so that for
users that care about monitoring performance cliffs, they can opt-in to log
when a fallback to interpreter mode was taken.
> Add optional logging to UnsafeProjection.create when it falls back to
> interpreted mode
> --------------------------------------------------------------------------------------
>
> Key: SPARK-25140
> URL: https://issues.apache.org/jira/browse/SPARK-25140
> Project: Spark
> Issue Type: Wish
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Kris Mok
> Priority: Minor
>
> SPARK-23711 implemented a nice graceful handling of allowing UnsafeProjection
> to fall back to an interpreter mode when codegen fails. That makes Spark much
> more usable even when codegen is unable to handle the given query.
> But in its current form, the fallback handling can also be a mystery in terms
> of performance cliffs. Users may be left wondering why a query runs fine with
> some expressions, but then with just one extra expression the performance
> goes 2x, 3x (or more) slower.
> It'd be nice to have optional logging of the fallback behavior, so that for
> users that care about monitoring performance cliffs, they can opt-in to log
> when a fallback to interpreter mode was taken. i.e. at
> https://github.com/apache/spark/blob/a40ffc656d62372da85e0fa932b67207839e7fde/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L183
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
