[
https://issues.apache.org/jira/browse/SPARK-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929334#comment-16929334
]
Liang-Chi Hsieh commented on SPARK-26205:
-----------------------------------------
[~cloud_fan] I ran a simple test, seems no failure happens?
val inSet = InSet(Literal(0), Set.empty)
checkEvaluation(inSet, false, row1)
I verified that it is codegen with genCodeWithSwitch.
> Optimize InSet expression for bytes, shorts, ints, dates
> --------------------------------------------------------
>
> Key: SPARK-26205
> URL: https://issues.apache.org/jira/browse/SPARK-26205
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Anton Okolnychyi
> Assignee: Anton Okolnychyi
> Priority: Major
> Fix For: 3.0.0
>
>
> {{In}} expressions are compiled into a sequence of if-else statements, which
> results in O\(n\) time complexity. {{InSet}} is an optimized version of
> {{In}}, which is supposed to improve the performance if the number of
> elements is big enough. However, {{InSet}} actually degrades the performance
> in many cases due to various reasons (benchmarks were created in SPARK-26203
> and solutions to the boxing problem are discussed in SPARK-26204).
> The main idea of this JIRA is to use Java {{switch}} statements to
> significantly improve the performance of {{InSet}} expressions for bytes,
> shorts, ints, dates. All {{switch}} statements are compiled into
> {{tableswitch}} and {{lookupswitch}} bytecode instructions. We will have
> O\(1\) time complexity if our case values are compact and {{tableswitch}} can
> be used. Otherwise, {{lookupswitch}} will give us O\(log n\). Our local
> benchmarks show that this logic is more than two times faster even on 500+
> elements than using primitive collections in {{InSet}} expressions. As Spark
> is using Scala {{HashSet}} right now, the performance gain will be is even
> bigger.
> See
> [here|https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-3.html#jvms-3.10]
> and
> [here|https://stackoverflow.com/questions/10287700/difference-between-jvms-lookupswitch-and-tableswitch]
> for more information.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
