[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13024 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218295471 Thanks, merging into master 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62754734
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -677,8 +677,9 @@ private[hive] class HiveClientImpl(
.asInstanceOf[Class[_ <:
org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]]]
private def toHiveFunction(f: CatalogFunction, db: String): HiveFunction
= {
-val resourceUris = f.resources.map { case (resourceType, resourcePath)
=>
- new ResourceUri(ResourceType.valueOf(resourceType.toUpperCase),
resourcePath)
+val resourceUris = f.resources.map { case resource =>
--- End diff --
I'm going to fix this myself when I merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62754706
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -677,8 +677,9 @@ private[hive] class HiveClientImpl(
.asInstanceOf[Class[_ <:
org.apache.hadoop.hive.ql.io.HiveOutputFormat[_, _]]]
private def toHiveFunction(f: CatalogFunction, db: String): HiveFunction
= {
-val resourceUris = f.resources.map { case (resourceType, resourcePath)
=>
- new ResourceUri(ResourceType.valueOf(resourceType.toUpperCase),
resourcePath)
+val resourceUris = f.resources.map { case resource =>
--- End diff --
by the way in the future when you only have one thing here you don't need
`case`. It's usually `case (a, b)` when there are multiple things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218288021 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58259/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218288018 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218287633 **[Test build #58259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58259/consoleFull)** for PR 13024 at commit [`f3842f2`](https://github.com/apache/spark/commit/f3842f2f31da0a97eed31415d361378411172f2b). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class FunctionResourceType(val resourceType: String)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218261494 **[Test build #58259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58259/consoleFull)** for PR 13024 at commit [`f3842f2`](https://github.com/apache/spark/commit/f3842f2f31da0a97eed31415d361378411172f2b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62733179
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/functionResources.scala
---
@@ -20,16 +20,24 @@ package org.apache.spark.sql.catalyst.catalog
import org.apache.spark.sql.AnalysisException
/** An trait that represents the type of a resourced needed by a function.
*/
-sealed trait FunctionResourceType
+sealed trait FunctionResourceType {
+ def resourceType: String
+}
-object JarResource extends FunctionResourceType
+object JarResource extends FunctionResourceType {
+ val resourceType = "jar"
--- End diff --
yeah, do that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62732815
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/functionResources.scala
---
@@ -20,16 +20,24 @@ package org.apache.spark.sql.catalyst.catalog
import org.apache.spark.sql.AnalysisException
/** An trait that represents the type of a resourced needed by a function.
*/
-sealed trait FunctionResourceType
+sealed trait FunctionResourceType {
+ def resourceType: String
+}
-object JarResource extends FunctionResourceType
+object JarResource extends FunctionResourceType {
+ val resourceType = "jar"
--- End diff --
If we make FunctionResourceType and abstract class
These could be written as `object JarResource extends
FunctionResourceType("jar)`
Any thoughts ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218257432 Just minor comments. LGTM overall. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62731779
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -636,10 +636,8 @@ class SessionCatalog(
* Loads resources such as JARs and Files for a function. Every resource
is represented
* by a tuple (resource type, resource uri).
*/
- def loadFunctionResources(resources: Seq[(String, String)]): Unit = {
-resources.foreach { case (resourceType, uri) =>
- val functionResource =
-
FunctionResource(FunctionResourceType.fromString(resourceType.toLowerCase), uri)
+ def loadFunctionResources(resources: Seq[FunctionResource]): Unit = {
+resources.foreach { case functionResource =>
functionResourceLoader.loadResource(functionResource)
--- End diff --
just do `resources.foreach(functionResourceLoader.loadResource)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62731677
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/functionResources.scala
---
@@ -20,16 +20,24 @@ package org.apache.spark.sql.catalyst.catalog
import org.apache.spark.sql.AnalysisException
/** An trait that represents the type of a resourced needed by a function.
*/
-sealed trait FunctionResourceType
+sealed trait FunctionResourceType {
+ def resourceType: String
+}
-object JarResource extends FunctionResourceType
+object JarResource extends FunctionResourceType {
+ val resourceType = "jar"
--- End diff --
you should explicitly do `override val`, here and other places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13024#discussion_r62731630
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/functionResources.scala
---
@@ -20,16 +20,24 @@ package org.apache.spark.sql.catalyst.catalog
import org.apache.spark.sql.AnalysisException
/** An trait that represents the type of a resourced needed by a function.
*/
-sealed trait FunctionResourceType
+sealed trait FunctionResourceType {
--- End diff --
just make it `abstract class FunctionResourceType(val resourceType)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218137264 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58230/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218137262 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218137050 **[Test build #58230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58230/consoleFull)** for PR 13024 at commit [`0de5f84`](https://github.com/apache/spark/commit/0de5f84978c28a34180a581dd185c395bb7b160b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218118749 **[Test build #58230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58230/consoleFull)** for PR 13024 at commit [`0de5f84`](https://github.com/apache/spark/commit/0de5f84978c28a34180a581dd185c395bb7b160b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218105273 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218105275 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58227/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218105268 **[Test build #58227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58227/consoleFull)** for PR 13024 at commit [`8c2f33c`](https://github.com/apache/spark/commit/8c2f33c45ed113f6e729818bf5584cc9d5c581a6). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218104344 **[Test build #58227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58227/consoleFull)** for PR 13024 at commit [`8c2f33c`](https://github.com/apache/spark/commit/8c2f33c45ed113f6e729818bf5584cc9d5c581a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
Github user techaddict commented on the pull request: https://github.com/apache/spark/pull/13024#issuecomment-218103693 cc: @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15249][SQL] Use FunctionResource instea...
GitHub user techaddict opened a pull request: https://github.com/apache/spark/pull/13024 [SPARK-15249][SQL] Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource ## What changes were proposed in this pull request? Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource see: TODO's here https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L36 https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala#L42 ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/techaddict/spark SPARK-15249 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13024.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13024 commit 8c2f33c45ed113f6e729818bf5584cc9d5c581a6 Author: Sandeep Singh Date: 2016-05-10T09:17:13Z [SPARK-15249][SQL] Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
