[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-04 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 Yeah. This refactoring was needed. Now it should be easier to make such changes. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-04 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21220 LGTM. Obviously shouldn't block this PR, but MicroBatchExecution is structured in a way that makes it hard to review changes like this. It seems like changing the condition under which

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-04 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz Answers to your questions. 1. We have already fixed those emptyDF optimizations. The optimization only quick in `df.isStreaming = false`, and emptyDFs generated by sources should be

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-04 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz Answers to your questions. 1. We have already fixed those emptyDF optimizations. The optimization only quick in `df.isStreaming = false`, and emptyDFs generated by sources should be

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90172/ Test PASSed. ---

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90172/testReport)** for PR 21220 at commit

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2891/

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90172/testReport)** for PR 21220 at commit

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90078/ Test FAILed. ---

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90078/testReport)** for PR 21220 at commit

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90080/ Test FAILed. ---

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90080/testReport)** for PR 21220 at commit

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2830/

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90080/testReport)** for PR 21220 at commit

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 @brkyvz @zsxwing @jose-torres --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2829/

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21220 jenkins retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21220 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2828/

[GitHub] spark issue #21220: [SPARK-24157][SS] Enabled no-data batches in MicroBatchE...

2018-05-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21220 **[Test build #90078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90078/testReport)** for PR 21220 at commit