[ https://issues.apache.org/jira/browse/SPARK-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-16898: ---------------------------------- Component/s: SQL > Adds argument type information for typed logical plan like MapElements, > TypedFilter, and AppendColumn > ----------------------------------------------------------------------------------------------------- > > Key: SPARK-16898 > URL: https://issues.apache.org/jira/browse/SPARK-16898 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Sean Zhong > Assignee: Sean Zhong > Priority: Minor > Fix For: 2.1.0 > > > Typed logical plan like MapElements, TypedFilter, and AppendColumn contains a > closure field: {{func: (T) => Boolean}}. For example class TypedFilter's > signature is: > {code} > case class TypedFilter( > func: AnyRef, > deserializer: Expression, > child: LogicalPlan) extends UnaryNode > {code} > From the above class signature, we cannot easily find: > 1. What is the input argument's type of the closure {{func}}? How do we know > which apply method to pick if there are multiple overloaded apply methods? > 2. What is the input argument's schema? > With this info, it is easier for us to define some custom optimizer rule to > translate these typed logical plan to more efficient implementation, like the > closure optimization idea in SPARK-14083. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org