Nikita Konda created SPARK-29029:
------------------------------------
Summary: PhysicalOperation.collectProjectsAndFilters should use
AttributeMap while substituting aliases
Key: SPARK-29029
URL: https://issues.apache.org/jira/browse/SPARK-29029
Project: Spark
Issue Type: Bug
Components: Optimizer
Affects Versions: 2.3.0
Reporter: Nikita Konda
We have a specific use case where in we are trying insert a custom logical
operator in our logical plan to avoid some of the Spark’s optimization rules.
However, we remove this logical operator as part of custom optimization rule
before we send this to SparkStrategies.
Analyzed plan:
{code:java}
[1] Project [userid#0]
+- [2] SubqueryAlias tmp6
+- [3] Project [videoid#47L, avebitrate#2, userid#0]
+- [4] Filter NOT (videoid#47L = cast(30 as bigint))
+- [5] SubqueryAlias tmp5
+- [6] CustomBarrier
+- [7] Project [videoid#47L, avebitrate#2, userid#0]
+- [8] Filter (avebitrate#2 < 10)
+- [9] SubqueryAlias tmp3
+- [10] Project [avebitrate#2, factorial(videoid#1) AS
videoid#47L, userid#0]
+- [11] SubqueryAlias tmp2
+- [12] Project [userid#0, videoid#1,
avebitrate#2]
+- [13] SubqueryAlias tmp1
+- [14] Project [userid#0, videoid#1,
avebitrate#2]
+- [15] SubqueryAlias views
+- [16]
Relation[userid#0,videoid#1,avebitrate#2]
{code}
Optimized Plan:
{code:java}
[1] Project [userid#0]
+- [2] Filter (isnotnull(videoid#47L) && NOT (videoid#47L = 30))
+- [3] Project [factorial(videoid#1) AS videoid#47L, userid#0]
+- [4] Filter (isnotnull(avebitrate#2) && (avebitrate#2 < 10))
+- [5] Relation[userid#0,videoid#1,avebitrate#2]
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
