[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15665 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15665#discussion_r85585863 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -388,16 +388,15 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) { partitionColumnNames = partitioningColumns.getOrElse(Nil), bucketSpec = getBucketSpec ) -val createCmd = CreateTable(tableDesc, mode, Some(df.logicalPlan)) -val cmd = if (tableDesc.partitionColumnNames.nonEmpty && +val result = df.sparkSession.sessionState.executePlan( + CreateTable(tableDesc, mode, Some(df.logicalPlan))) --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15665#discussion_r85577862 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -254,7 +254,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) { ) } -df.sparkSession.sessionState.executePlan( +df.sparkSession.sessionState. --- End diff -- Some github text editing gone wrong... fixed it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15665#discussion_r85490391 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -254,7 +254,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) { ) } -df.sparkSession.sessionState.executePlan( +df.sparkSession.sessionState. --- End diff -- what's going on here? a dot ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15665#discussion_r85466277 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala --- @@ -388,16 +388,15 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) { partitionColumnNames = partitioningColumns.getOrElse(Nil), bucketSpec = getBucketSpec ) -val createCmd = CreateTable(tableDesc, mode, Some(df.logicalPlan)) -val cmd = if (tableDesc.partitionColumnNames.nonEmpty && +val result = df.sparkSession.sessionState.executePlan( + CreateTable(tableDesc, mode, Some(df.logicalPlan))) --- End diff -- yea, we don't need the `result` variable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15665: [SPARK-18146] [SQL] Avoid using Union to chain to...
GitHub user ericl opened a pull request: https://github.com/apache/spark/pull/15665 [SPARK-18146] [SQL] Avoid using Union to chain together create table and repair partition commands ## What changes were proposed in this pull request? The behavior of union is not well defined here. It is safer to explicitly execute these commands in order. The other use of `Union` in this way will be removed by https://github.com/apache/spark/pull/15633 ## How was this patch tested? Existing tests. cc @yhuai You can merge this pull request into a Git repository by running: $ git pull https://github.com/ericl/spark spark-18146 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15665.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15665 commit 11fc8c4d55293e0e01549d9339ab9e6139b1a29e Author: Eric LiangDate: 2016-10-27T23:05:15Z avoid union --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org