[ https://issues.apache.org/jira/browse/SPARK-33915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257647#comment-17257647 ]
Ted Yu commented on SPARK-33915: -------------------------------- Here is sample code for capturing the column and fields in downstream PredicatePushDown.scala {code} private val JSONCapture = "`GetJsonObject\\((.*),(.*)\\)`".r private def transformGetJsonObject(p: Predicate): Predicate = { val eq = p.asInstanceOf[sources.EqualTo] eq.attribute match { case JSONCapture(column,field) => val colName = column.toString.split("#")(0) val names = field.toString.split("\\.").foldLeft(List[String]()){(z, n) => z :+ "->'"+n+"'" } sources.EqualTo(colName + names.slice(1, names.size).mkString(""), eq.value).asInstanceOf[Predicate] case _ => sources.EqualTo("foo", "bar").asInstanceOf[Predicate] } } {code} > Allow json expression to be pushable column > ------------------------------------------- > > Key: SPARK-33915 > URL: https://issues.apache.org/jira/browse/SPARK-33915 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.0.1 > Reporter: Ted Yu > Assignee: Apache Spark > Priority: Major > > Currently PushableColumnBase provides no support for json / jsonb expression. > Example of json expression: > {code} > get_json_object(phone, '$.code') = '1200' > {code} > If non-string literal is part of the expression, the presence of cast() would > complicate the situation. > Implication is that implementation of SupportsPushDownFilters doesn't have a > chance to perform pushdown even if third party DB engine supports json > expression pushdown. > This issue is for discussion and implementation of Spark core changes which > would allow json expression to be recognized as pushable column. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org