[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once
[
https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510917#comment-16510917
]
LANDAIS Christophe commented on SPARK-23486:
Hi,
What is the process to have the issue corrected and submitted ?
Thanks and BR,
Christophe
> LookupFunctions should not check the same function name more than once
> --
>
> Key: SPARK-23486
> URL: https://issues.apache.org/jira/browse/SPARK-23486
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.1, 2.3.0
>Reporter: Cheng Lian
>Priority: Major
> Labels: starter
>
> For a query invoking the same function multiple times, the current
> {{LookupFunctions}} rule performs a check for each invocation. For users
> using Hive metastore as external catalog, this issues unnecessary metastore
> accesses and can slow down the analysis phase quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once
[
https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479539#comment-16479539
]
Marcelo Vanzin commented on SPARK-23486:
I removed the target versions since the PR seems stalled in review.
> LookupFunctions should not check the same function name more than once
> --
>
> Key: SPARK-23486
> URL: https://issues.apache.org/jira/browse/SPARK-23486
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.1, 2.3.0
>Reporter: Cheng Lian
>Priority: Major
> Labels: starter
>
> For a query invoking the same function multiple times, the current
> {{LookupFunctions}} rule performs a check for each invocation. For users
> using Hive metastore as external catalog, this issues unnecessary metastore
> accesses and can slow down the analysis phase quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once
[
https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394422#comment-16394422
]
Apache Spark commented on SPARK-23486:
--
User 'kevinyu98' has created a pull request for this issue:
https://github.com/apache/spark/pull/20795
> LookupFunctions should not check the same function name more than once
> --
>
> Key: SPARK-23486
> URL: https://issues.apache.org/jira/browse/SPARK-23486
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.1, 2.3.0
>Reporter: Cheng Lian
>Priority: Major
> Labels: starter
>
> For a query invoking the same function multiple times, the current
> {{LookupFunctions}} rule performs a check for each invocation. For users
> using Hive metastore as external catalog, this issues unnecessary metastore
> accesses and can slow down the analysis phase quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once
[
https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373851#comment-16373851
]
kevin yu commented on SPARK-23486:
--
Hello [~lian cheng]: I think the easy way is to build a hash map around the
LookupFunctions, if the function exists in the external catalog, put into the
hash map for the first time, next time when call the LookupFunctions, first
check the hash map to avoid the metastore accesses, does this approach look ok
to you? If you think it is ok, I can provide a pr for reviewing. Thanks.
Another approach is to cache the external catalog functions in the share state,
many queries can use, but it will be more involved to do the invalidation.
> LookupFunctions should not check the same function name more than once
> --
>
> Key: SPARK-23486
> URL: https://issues.apache.org/jira/browse/SPARK-23486
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.1, 2.3.0
>Reporter: Cheng Lian
>Priority: Major
> Labels: starter
>
> For a query invoking the same function multiple times, the current
> {{LookupFunctions}} rule performs a check for each invocation. For users
> using Hive metastore as external catalog, this issues unnecessary metastore
> accesses and can slow down the analysis phase quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once
[
https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372285#comment-16372285
]
Cheng Lian commented on SPARK-23486:
Please refer to [this
comment|https://issues.apache.org/jira/browse/SPARK-19737?focusedCommentId=16371377=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16371377]
for more details.
> LookupFunctions should not check the same function name more than once
> --
>
> Key: SPARK-23486
> URL: https://issues.apache.org/jira/browse/SPARK-23486
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.1, 2.3.0
>Reporter: Cheng Lian
>Priority: Major
>
> For a query invoking the same function multiple times, the current
> {{LookupFunctions}} rule performs a check for each invocation. For users
> using Hive metastore as external catalog, this issues unnecessary metastore
> accesses and can slow down the analysis phase quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
