[
https://issues.apache.org/jira/browse/SPARK-16203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349582#comment-15349582
]
Max Moroz commented on SPARK-16203:
-----------------------------------
Hive SQL syntax allows the return value from a function to be an array; for
example split does it. I understand overloading the existing name may be
confusing, but would it be inappropriate to add another function (like
regexp_extract_n)?
If I'm misunderstanding something, and parsing something like a web log with
DataFrame API is already perfectly efficient, I would not think it's worth
doing. But I don't think it's currently possible to do efficient parsing (the
best solution I'm aware of is regexp_replace, and then split - perhaps the
optimizer manages to optimize away the unnecessary insertion of new characters,
but I don't think so?).
> regexp_extract to return an ArrayType(StringType())
> ---------------------------------------------------
>
> Key: SPARK-16203
> URL: https://issues.apache.org/jira/browse/SPARK-16203
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.0.0
> Reporter: Max Moroz
> Priority: Minor
>
> regexp_extract only returns a single matched group. If (as if often the case
> - e.g., web log parsing) we need to parse the entire line and get all the
> groups, we'll need to call it as many times as there are groups.
> It's only a minor annoyance syntactically.
> But unless I misunderstand something, it would be very inefficient. (How
> would Spark know not to do multiple pattern matching operations, when only
> one is needed? Or does the optimizer actually check whether the patterns are
> identical, and if they are, avoid the repeated regex matching operations??)
> Would it be possible to have it return an array when the index is not
> specified (defaulting to None)?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
