[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/13653 I am merging this for now. Please keep reviewing this and I will update them in #13673 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60519/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60519/consoleFull)** for PR 13653 at commit [`29ca23b`](https://github.com/apache/spark/commit/29ca23b74c16440dbb1e440423615357ec8015bf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60519/consoleFull)** for PR 13653 at commit [`29ca23b`](https://github.com/apache/spark/commit/29ca23b74c16440dbb1e440423615357ec8015bf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60516/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60516/consoleFull)** for PR 13653 at commit [`536c25e`](https://github.com/apache/spark/commit/536c25e00b154a467279ab0055ae98002bd70847). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60516/consoleFull)** for PR 13653 at commit [`536c25e`](https://github.com/apache/spark/commit/536c25e00b154a467279ab0055ae98002bd70847). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/13653 @marmbrus Overall, I have changed the following. - Renamed writeStream.save() to writeStream.start() - Refactored writeStream.foreach() to not start the query. instead writeStream.foreach().start() starts the query - Removed writeStream.parquet(path) for now. Its good to have a single method `start` actually start the background query. experience says that its much easier read code and debug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/13653 Overall looks pretty good. Feel free to merge after addressing comments / passing tests to avoid more conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60461/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60461/consoleFull)** for PR 13653 at commit [`bbfff70`](https://github.com/apache/spark/commit/bbfff70b0bac8b2c33ab1eb736405759bb391633). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60461/consoleFull)** for PR 13653 at commit [`bbfff70`](https://github.com/apache/spark/commit/bbfff70b0bac8b2c33ab1eb736405759bb391633). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60460/consoleFull)** for PR 13653 at commit [`e118631`](https://github.com/apache/spark/commit/e118631fd5bbea962932a812addae21c89f6f4f2). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class DataStreamReader(object):` * `class DataStreamWriter(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60460/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60460/consoleFull)** for PR 13653 at commit [`e118631`](https://github.com/apache/spark/commit/e118631fd5bbea962932a812addae21c89f6f4f2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13653 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60452/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60452/consoleFull)** for PR 13653 at commit [`a59498b`](https://github.com/apache/spark/commit/a59498bbe86609bc206de9b229052f35071049cb). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13653: [SPARK-15933][SQL][STREAMING] Refactored DF reader-write...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13653 **[Test build #60452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60452/consoleFull)** for PR 13653 at commit [`a59498b`](https://github.com/apache/spark/commit/a59498bbe86609bc206de9b229052f35071049cb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
