[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user skparkes commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175775385 Thanks a bunch guys, this will be a big help. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175770352 I also merged it in branch 1.5 and branch 1.6 since it is very isolated bug fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8969 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175768840 LGTM. I am merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175370156 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50160/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175370154 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175369175 **[Test build #50160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50160/consoleFull)** for PR 8969 at commit [`5524a92`](https://github.com/apache/spark/commit/5524a92e738d034342be6b8aa61116ef50c520c0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175341869 **[Test build #50160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50160/consoleFull)** for PR 8969 at commit [`5524a92`](https://github.com/apache/spark/commit/5524a92e738d034342be6b8aa61116ef50c520c0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175338382 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-175338290 Ping @yhuai, is this ready for merging? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user jasoncl commented on a diff in the pull request:
https://github.com/apache/spark/pull/8969#discussion_r49683600
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -229,6 +231,9 @@ class MetadataBuilder {
this
}
+ /** Puts a null. */
+ def putNull(key: String): this.type = put(key, null)
--- End diff --
The method is created for code organization and clarity purpose. This way
it can be easily reused in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-171387160 @jasoncl Can you do a quick update if my comment makes sense? Then, we will get it merged. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8969#discussion_r49628004
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -229,6 +231,9 @@ class MetadataBuilder {
this
}
+ /** Puts a null. */
+ def putNull(key: String): this.type = put(key, null)
--- End diff --
Looks like not worth adding a method since it is just used once?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user jasoncl commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150701775 The test build #44243 for commit 978fdc7 shows that the patch does not merge cleanly. But that is not my latest commit. My latest commit 5524a92 has passed the test build and will merge cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150696360 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150696362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44243/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150696226 [Test build #44243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44243/console) for PR 8969 at commit [`978fdc7`](https://github.com/apache/spark/commit/978fdc7efce61b41973013b7fe2bf2ba7e7df596). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150682844 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44246/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150682712 **[Test build #44246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44246/consoleFull)** for PR 8969 at commit [`5524a92`](https://github.com/apache/spark/commit/5524a92e738d034342be6b8aa61116ef50c520c0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150682842 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150662280 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44245/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150662279 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150660369 **[Test build #44246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44246/consoleFull)** for PR 8969 at commit [`5524a92`](https://github.com/apache/spark/commit/5524a92e738d034342be6b8aa61116ef50c520c0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150659611 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150659658 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150658487 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150658469 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150656437 [Test build #44243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44243/consoleFull) for PR 8969 at commit [`978fdc7`](https://github.com/apache/spark/commit/978fdc7efce61b41973013b7fe2bf2ba7e7df596). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150654205 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-150654189 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149064189 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149064191 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43900/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149064113 [Test build #43900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43900/console) for PR 8969 at commit [`6bb4fcc`](https://github.com/apache/spark/commit/6bb4fcc09c85ba1e039113d5a0172d87ea1d590a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149047792 This seems good overall, but one high-level question: why store `None` on the Java side instead of `null`? I'm just wondering whether mapping it back to Java `null` would make more sense. I guess it doesn't make a difference from Python's point of view, since Python's view of the metadata is preserved roundtrip, but it could hypothetically matter if Python is trying to set metadata which is then consumed by Java library code (for example). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149047716 [Test build #43900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43900/consoleFull) for PR 8969 at commit [`6bb4fcc`](https://github.com/apache/spark/commit/6bb4fcc09c85ba1e039113d5a0172d87ea1d590a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8969#discussion_r42326412
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -228,6 +232,8 @@ class MetadataBuilder {
map ++= metadata.map
this
}
+ /** Puts a Long. */
--- End diff --
This comment is incorrect. Also, please leave an additional line of
whitespace before this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149046930 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149046921 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-149046745 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user skparkes commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-145189331 Thanks for doing this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
GitHub user jasoncl opened a pull request:
https://github.com/apache/spark/pull/8969
[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with
`None` triggers cryptic failure
The error message is now changed from "Do not support type class
scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be
more informative about what is not supported. Also, StructType metadata now
handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to
tests.py to show the fix works.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jasoncl/spark SPARK-10847
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8969.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8969
commit 362778c2f9fd5a30959e6451017a1bdf428cb272
Author: Jason Lee
Date: 2015-10-02T18:14:53Z
Added support for JNull in Metadata. Error message now displays the actual
unsupported type in the tuple instead of just tuple2
commit 6bb4fcc09c85ba1e039113d5a0172d87ea1d590a
Author: Jason Lee
Date: 2015-10-02T21:51:05Z
fix scala style
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10847] [SQL] [PySpark] Pyspark - DataFr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8969#issuecomment-145177726 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
