[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user chiyingyunhua commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-113010184 请é®ï¼æ使ç¨spark1.3.1çsparksqlä¸dataframeçä¸ä¸ªsaveastableæ¹æ³åäºä¸ä¸ªdataframeï¼ä½æ¯ç¨hive读çæ¶ååºéäºFailed with exception java.io.IOException:java.io.IOException: hdfs://namenode71:8020/user/hive/warehouse/zz5/part-0 not a SequenceFileãè¿ä¸ªæä¹ç ´ï¼1.2æ¯å¯ä»¥ç --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101060308 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101060293 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101061693 LGTM. I will merge it after Jenkins comes back happy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101060254 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101079133 FYI I submitted a small patch on top of this to add a link to a jira ticket: https://github.com/apache/spark/pull/6067 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101060643 [Test build #32428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32428/consoleFull) for PR 5654 at commit [`386f407`](https://github.com/apache/spark/commit/386f407bad7a1b93c70194af48a34fd8eb3af229). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101077524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32428/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101077511 [Test build #32428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32428/consoleFull) for PR 5654 at commit [`386f407`](https://github.com/apache/spark/commit/386f407bad7a1b93c70194af48a34fd8eb3af229). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-101077523 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5654 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-96766750 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95666418 [Test build #30856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30856/consoleFull) for PR 5654 at commit [`00bc819`](https://github.com/apache/spark/commit/00bc819ba948dd29d78251c7d59089ce1116bc2e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/5654#discussion_r28994140 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -1085,6 +1085,9 @@ class DataFrame private[sql]( * there is no notion of a persisted catalog in a standard SQL context. Instead you can write * an RDD out to a parquet file, and then register that file as a table. This table can then * be the target of an `insertInto`. + * + * Also note that while this function can persist the table metadata into Hive's metastore, + * the table will NOT be accessible from Hive. --- End diff -- There are multiple saveAsTable functions. Do you mind updating them as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95665258 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user phatak-dev commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95798250 Added for other methods also. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95689131 [Test build #30856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30856/consoleFull) for PR 5654 at commit [`00bc819`](https://github.com/apache/spark/commit/00bc819ba948dd29d78251c7d59089ce1116bc2e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FreqItemset(namedtuple(FreqItemset, [items, freq])):` * `protected[sql] abstract class AtomicType extends DataType ` * `abstract class NumericType extends AtomicType ` * ` class Encoder[T : AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Decoder[T : AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])` * ` class Encoder[T : AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Decoder[T : AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])` * ` class Encoder[T : AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Decoder[T : AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95689145 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30856/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
GitHub user phatak-dev opened a pull request: https://github.com/apache/spark/pull/5654 [SPARK-7084] improve saveAsTable documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/phatak-dev/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5654.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5654 commit 866e8dfa32501b1853cbe3cb52fc569ec34a80c2 Author: madhukar phatak@gmail.com Date: 2015-04-23T08:10:47Z [SPARK-7084] improve saveAsTable documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/5654#discussion_r28945850 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala --- @@ -1085,6 +1085,9 @@ class DataFrame private[sql]( * there is no notion of a persisted catalog in a standard SQL context. Instead you can write * an RDD out to a parquet file, and then register that file as a table. This table can then * be the target of an `insertInto`. + * + * Also note that this doesn't create a hive table, but instead creates a Spark data source table. --- End diff -- Maybe say Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95486689 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7084] improve saveAsTable documentation
Github user phatak-dev commented on the pull request: https://github.com/apache/spark/pull/5654#issuecomment-95522943 Updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
