[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16645 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16645#discussion_r97003427 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala --- @@ -62,15 +62,17 @@ private[hive] class HiveSessionState(sparkSession: SparkSession) override val extendedResolutionRules = catalog.ParquetConversions :: catalog.OrcConversions :: --- End diff -- These two rules need `MetastoreRelation`. Ideally, they should be after the rule `FindHiveSerdeTable`. I am fine to keep it if we plan to move it into optimizer rules. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16645#discussion_r97003206 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala --- @@ -62,15 +62,17 @@ private[hive] class HiveSessionState(sparkSession: SparkSession) override val extendedResolutionRules = catalog.ParquetConversions :: catalog.OrcConversions :: --- End diff -- do they need to? Eventually they will be optimizer rules. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16645#discussion_r96976865 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala --- @@ -62,15 +62,17 @@ private[hive] class HiveSessionState(sparkSession: SparkSession) override val extendedResolutionRules = catalog.ParquetConversions :: catalog.OrcConversions :: --- End diff -- How about moving the rule catalog.ParquetConversions and catalog.OrcConversions at the beginning of the batch `postHocResolutionRules `? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16645#discussion_r96976789 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala --- @@ -62,15 +62,17 @@ private[hive] class HiveSessionState(sparkSession: SparkSession) override val extendedResolutionRules = catalog.ParquetConversions :: catalog.OrcConversions :: -AnalyzeCreateTable(sparkSession) :: -PreprocessTableInsertion(conf) :: -DataSourceAnalysis(conf) :: new DetermineHiveSerde(conf) :: -new HiveAnalysis(sparkSession) :: new FindDataSourceTable(sparkSession) :: new FindHiveSerdeTable(sparkSession) :: new ResolveDataSource(sparkSession) :: Nil + override val postHocResolutionRules = +AnalyzeCreateTable(sparkSession) :: +PreprocessTableInsertion(conf) :: +DataSourceAnalysis(conf) :: +new HiveAnalysis(sparkSession) :: Nil --- End diff -- How about moving the rule `catalog.ParquetConversions` and `catalog.OrcConversions` at the beginning of this batch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16645: [SPARK-19290][SQL] add a new extending interface ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16645#discussion_r96941832 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -106,6 +106,13 @@ class Analyzer( */ val extendedResolutionRules: Seq[Rule[LogicalPlan]] = Nil + /** + * Override to provide rules to do post-hoc resolution. Note that these rules will be executed + * in an individual bach. This batch is run right after the normal resolution batch and execute --- End diff -- `bach` -> `batch` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
