[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 Oh, BTW it seems `read.df` also seems not allowing this? I will try to test and fix here together if so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 @felixcheung , I usually don't like to answer by quote but let me do this just to clarify. > Hmm, should we hold till 12601 is merged then? Seems like we shouldn't allow this unless internal datasources are supporting this more broadly. As omitting `path` is what the datasource interface allows, maybe, it'd be just okay to test if it goes through JVM fine. Also, I worry if I can easily add a test for JDBC datasource within SparkR. If it can be easily done, I am also happy to hold this. > Also, before the path parameter type is in the signature, ie. > > ``` > write.df(df, c(1, 2)) > ``` > > Would error with some descriptive error, with this change it would get some JVM exception which seems to degrade the experience a bit. Yeap, I could add some type checks > Similarly for the path not specified case java.lang.IllegalArgumentException - we generally try to avoid JVM exception showing up if possible. Also, yes. Maybe, we could avoid the direct JVM message after catching this and make it pretty within R just like PySpark does[1]. (although I am not sure if it sounds good in R). > Could you add checks to path for these cases and give more descriptive messages? Sure, I will try to address the points. [1]https://github.com/apache/spark/blob/9a5071996b968148f6b9aba12e0d3fe888d9acd8/python/pyspark/sql/utils.py#L64-L80 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15231 Hmm, should we hold till 12601 is merged then? Seems like we shouldn't allow this unless internal datasources are supporting this more broadly. Also, before the path parameter type is in the signature, ie. ``` write.df(df, c(1, 2)) ``` Would error with some descriptive error, with this change it would get some JVM exception which seems to degrade the experience a bit. Could you add checks for `path` and give more descriptive message? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65880/consoleFull)** for PR 15231 at commit [`c2a64db`](https://github.com/apache/spark/commit/c2a64dba42d68561c23583ca8bbb79aa4edfb707). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65880/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 @felixcheung Yeap, I just added. As we don't currently have an internal datasource allowing `DataFrameWriter.save()` without a path, I just added a test with CSV data source to make sure it throws an exception from Spark side (not from SparkR). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65880/consoleFull)** for PR 15231 at commit [`c2a64db`](https://github.com/apache/spark/commit/c2a64dba42d68561c23583ca8bbb79aa4edfb707). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15231 Let's add some tests for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65868/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65868/consoleFull)** for PR 15231 at commit [`2d76e7c`](https://github.com/apache/spark/commit/2d76e7c54086dca703f4ffc180855c837bbc74e6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65868/consoleFull)** for PR 15231 at commit [`2d76e7c`](https://github.com/apache/spark/commit/2d76e7c54086dca703f4ffc180855c837bbc74e6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65866/consoleFull)** for PR 15231 at commit [`41611e5`](https://github.com/apache/spark/commit/41611e594c4e5f8e6aedb2846abfda74dac84a80). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65866/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65866/consoleFull)** for PR 15231 at commit [`41611e5`](https://github.com/apache/spark/commit/41611e594c4e5f8e6aedb2846abfda74dac84a80). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65865/consoleFull)** for PR 15231 at commit [`41611e5`](https://github.com/apache/spark/commit/41611e594c4e5f8e6aedb2846abfda74dac84a80). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15231 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65865/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 FWIW - Python API seems having `path` as optional[1] and Scala API seems doing this so as well[2]. [1]https://github.com/apache/spark/blob/master/python/pyspark/sql/readwriter.py#L521 [2]https://github.com/apache/spark/blob/b50b34f5611a1f182ba9b6eaf86c666bbd9f9eb0/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L189-L214 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15231 **[Test build #65865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65865/consoleFull)** for PR 15231 at commit [`41611e5`](https://github.com/apache/spark/commit/41611e594c4e5f8e6aedb2846abfda74dac84a80). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15231: [SPARK-17658][SPARKR] write.df API taking path optionall...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15231 cc @felixcheung and @shivaram --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
