[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user carsonwang commented on the issue: https://github.com/apache/spark/pull/20303 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20420: [SPARK-22916][SQL][FOLLOW-UP] Update the Description of ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20420 **[Test build #86753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86753/testReport)** for PR 20420 at commit [`ac8b1bc`](https://github.com/apache/spark/commit/ac8b1bcea76368ff27de30b1696694dadde7e658). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86756/testReport)** for PR 20404 at commit [`dd1c991`](https://github.com/apache/spark/commit/dd1c9912101389983011b47292da08eff0f43cdc). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86756/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20397 **[Test build #86754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86754/testReport)** for PR 20397 at commit [`b7fbd9d`](https://github.com/apache/spark/commit/b7fbd9d54c6d1725905700a9f8cfbab4f550be73). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20402 **[Test build #86751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86751/testReport)** for PR 20402 at commit [`d0cfbd0`](https://github.com/apache/spark/commit/d0cfbd0b1975f0b35537eb2c573fe758048319f7). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86754/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20402 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20420: [SPARK-22916][SQL][FOLLOW-UP] Update the Description of ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20420 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86753/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20420: [SPARK-22916][SQL][FOLLOW-UP] Update the Description of ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20420 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20402 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86751/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20303 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/326/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20303 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20404 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/327/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86758/testReport)** for PR 20404 at commit [`dd1c991`](https://github.com/apache/spark/commit/dd1c9912101389983011b47292da08eff0f43cdc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20303 **[Test build #86757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86757/testReport)** for PR 20303 at commit [`9a1301f`](https://github.com/apache/spark/commit/9a1301f1981a0725364b86c2aafe72a87110e7a3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/328/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86759/testReport)** for PR 20404 at commit [`ec94c05`](https://github.com/apache/spark/commit/ec94c054a8e0abf2d13434556d0434b4690590d3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86759/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86759/testReport)** for PR 20404 at commit [`ec94c05`](https://github.com/apache/spark/commit/ec94c054a8e0abf2d13434556d0434b4690590d3). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86760/testReport)** for PR 20404 at commit [`d9f77ea`](https://github.com/apache/spark/commit/d9f77eaadecedc115dd2dc64843761a3bb08d46e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86760/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86760/testReport)** for PR 20404 at commit [`d9f77ea`](https://github.com/apache/spark/commit/d9f77eaadecedc115dd2dc64843761a3bb08d46e). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/329/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20375: [SPARK-23199][SQL]improved Removes repetition from group...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20375 **[Test build #86761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86761/testReport)** for PR 20375 at commit [`e1ff90e`](https://github.com/apache/spark/commit/e1ff90ed613372163cd93330b1d9e33f9873c6de). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/330/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/331/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20397 **[Test build #86763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86763/testReport)** for PR 20397 at commit [`e570fca`](https://github.com/apache/spark/commit/e570fca9b16d320372869039305a12519e9b9b62). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86762/testReport)** for PR 20404 at commit [`1ed62ef`](https://github.com/apache/spark/commit/1ed62ef1dbc2c46f2d21eb6147418a86bd59c55c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkS...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20404#discussion_r164361700 --- Diff: python/pyspark/sql/session.py --- @@ -213,7 +213,12 @@ def __init__(self, sparkContext, jsparkSession=None): self._jsc = self._sc._jsc self._jvm = self._sc._jvm if jsparkSession is None: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) +if self._jvm.SparkSession.getDefaultSession().isDefined() \ +and not self._jvm.SparkSession.getDefaultSession().get() \ +.sparkContext().isStopped(): --- End diff -- I guess this change at 4ba3aa2af1b7bbc69575c14fffed18d5f1f90d53 is enough to fix the previous test failure (`ERROR: test_sparksession_with_stopped_sparkcontext (pyspark.sql.tests.SQLTests2)`) and we can revert moving `self._jvm.SparkSession.clearDefaultSession()` to `SparkContext.stop()` at 0319fa5c0527f68f3a3862afbbfd1b708f1d307d now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkS...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20404#discussion_r164362178 --- Diff: python/pyspark/sql/session.py --- @@ -225,6 +230,9 @@ def __init__(self, sparkContext, jsparkSession=None): if SparkSession._instantiatedSession is None \ or SparkSession._instantiatedSession._sc._jsc is None: SparkSession._instantiatedSession = self +if self._jvm.SparkSession.getDefaultSession().isEmpty() \ +or not jsparkSession.equals(self._jvm.SparkSession.getDefaultSession().get()): + self._jvm.SparkSession.setDefaultSession(self._jsparkSession) --- End diff -- Maybe we can simply overwrite the default session. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86758/testReport)** for PR 20404 at commit [`dd1c991`](https://github.com/apache/spark/commit/dd1c9912101389983011b47292da08eff0f43cdc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86758/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkS...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/20404#discussion_r164364286 --- Diff: python/pyspark/sql/session.py --- @@ -213,7 +213,12 @@ def __init__(self, sparkContext, jsparkSession=None): self._jsc = self._sc._jsc self._jvm = self._sc._jvm if jsparkSession is None: -jsparkSession = self._jvm.SparkSession(self._jsc.sc()) +if self._jvm.SparkSession.getDefaultSession().isDefined() \ +and not self._jvm.SparkSession.getDefaultSession().get() \ +.sparkContext().isStopped(): --- End diff -- I think so. I will do it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkS...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/20404#discussion_r164364573 --- Diff: python/pyspark/sql/session.py --- @@ -225,6 +230,9 @@ def __init__(self, sparkContext, jsparkSession=None): if SparkSession._instantiatedSession is None \ or SparkSession._instantiatedSession._sc._jsc is None: SparkSession._instantiatedSession = self +if self._jvm.SparkSession.getDefaultSession().isEmpty() \ +or not jsparkSession.equals(self._jvm.SparkSession.getDefaultSession().get()): + self._jvm.SparkSession.setDefaultSession(self._jsparkSession) --- End diff -- @felixcheung has concern about simply overwriting the default session. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20408 @ajbozarth What about `Active except for Stages: [...]`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20295: [WIP][SPARK-23011] Support alternative function f...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20295#discussion_r164347701 --- Diff: python/pyspark/sql/udf.py --- @@ -54,7 +54,7 @@ def _create_udf(f, returnType, evalType): "Instead, create a 1-arg pandas_udf and ignore the arg in your function." ) -if evalType == PythonEvalType.SQL_PANDAS_GROUP_MAP_UDF and len(argspec.args) != 1: +if evalType == PythonEvalType.SQL_PANDAS_GROUP_MAP_UDF and len(argspec.args) not in (1, 2): raise ValueError( "Invalid function: pandas_udfs with function type GROUP_MAP " "must take a single arg that is a pandas DataFrame." --- End diff -- We should update the error message here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20408#discussion_r164368745
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -25,9 +25,13 @@ function getThreadDumpEnabled() {
return threadDumpEnabled;
}
-function formatStatus(status, type) {
+function formatStatus(status, type, row) {
--- End diff --
Ok, done.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20408 **[Test build #86764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86764/testReport)** for PR 20408 at commit [`2c4c892`](https://github.com/apache/spark/commit/2c4c8927e2f56dbf09f47016a24d2daa4c5f6823). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86762/testReport)** for PR 20404 at commit [`1ed62ef`](https://github.com/apache/spark/commit/1ed62ef1dbc2c46f2d21eb6147418a86bd59c55c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86762/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20393: [SPARK-23207][SQL] Shuffle+Repartition on a DataFrame co...
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20393 @mridulm one approach that Xingbo is looking into (independently of https://github.com/apache/spark/pull/20414) is to have the `ShuffleBlockFetcherIterator` remember the order of blocks it fetches and store them in that order. Given that the blocks will still be fetched in parallel, depending on the available buffer size, we'll then have to spill some out-of-order blocks on disk in order to avoid OOMs on the receiver (similar to https://github.com/apache/spark/pull/16989). While this would still regress performance, it might be better than the current local sort based fix. Note that I'm not arguing against the fact that hash partitioning would be the "best" fix in terms of performance, but it'd then defeat the purpose of repartition (due to skew). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkS...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20404#discussion_r164372440 --- Diff: python/pyspark/sql/session.py --- @@ -225,6 +230,9 @@ def __init__(self, sparkContext, jsparkSession=None): if SparkSession._instantiatedSession is None \ or SparkSession._instantiatedSession._sc._jsc is None: SparkSession._instantiatedSession = self +if self._jvm.SparkSession.getDefaultSession().isEmpty() \ +or not jsparkSession.equals(self._jvm.SparkSession.getDefaultSession().get()): + self._jvm.SparkSession.setDefaultSession(self._jsparkSession) --- End diff -- I might miss something, but I guess @felixcheung's concern was fixed by checking if the default session is defined and not stopped so we can put the valid session or the same session from JVM without checking anymore. But I'm okay to leave it as it is as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/332/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86765/testReport)** for PR 20404 at commit [`e5f4b58`](https://github.com/apache/spark/commit/e5f4b580d3ecf69f30638ca1bc36073e132c489e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86765/testReport)** for PR 20404 at commit [`e5f4b58`](https://github.com/apache/spark/commit/e5f4b580d3ecf69f30638ca1bc36073e132c489e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20404 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86765/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20421: [SPARK-23112][DOC] Update ML migration guide with...
GitHub user MLnick opened a pull request: https://github.com/apache/spark/pull/20421 [SPARK-23112][DOC] Update ML migration guide with breaking changes. Add breaking change note to ML migration guide. ## How was this patch tested? Doc only You can merge this pull request into a Git repository by running: $ git pull https://github.com/MLnick/spark SPARK-23112-ml-guide Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20421 commit 184d1133d1557211d8bb309883af60613c3e77fb Author: Nick Pentreath Date: 2018-01-23T14:05:51Z Add highlights and migration guide for 2.3 commit cebc49613a179b318469d529df6d0a756ec59a2d Author: Nick Pentreath Date: 2018-01-29T09:47:42Z Merge branch 'master' into SPARK-23112-ml-guide commit b6fcf15dec51575bef0b1df787d57ba36b26ef63 Author: Nick Pentreath Date: 2018-01-29T10:02:24Z Update migration guide with breaking changes --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20332: [SPARK-23138][ML][DOC] Multiclass logistic regres...
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/20332#discussion_r164384660 --- Diff: docs/ml-classification-regression.md --- @@ -111,10 +110,9 @@ Continuing the earlier example: [`LogisticRegressionTrainingSummary`](api/java/org/apache/spark/ml/classification/LogisticRegressionTrainingSummary.html) provides a summary for a [`LogisticRegressionModel`](api/java/org/apache/spark/ml/classification/LogisticRegressionModel.html). -Currently, only binary classification is supported and the -summary must be explicitly cast to -[`BinaryLogisticRegressionTrainingSummary`](api/java/org/apache/spark/ml/classification/BinaryLogisticRegressionTrainingSummary.html). -Support for multiclass model summaries will be added in the future. +In the case of binary classification, certain additional metrics are --- End diff -- What do you mean exactly? Do you propose to list the metrics in the user guide? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...
GitHub user yaooqinn opened a pull request: https://github.com/apache/spark/pull/20422 [SPARK-23253][Core][Shuffle]Only write shuffle temporary index file when there is not an existing one ## What changes were proposed in this pull request? Shuffle Index temporay file is used for atomic creating shuffle index file, it is not needed when the index file already exists after another attempts of same task had it done. ## How was this patch tested? exitsting ut cc @squito You can merge this pull request into a Git repository by running: $ git pull https://github.com/yaooqinn/spark SPARK-23253 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20422 commit 98ea6a742143da803eb728c352e7424f504fabba Author: Kent Yao Date: 2018-01-29T10:11:50Z Only write shuffle temporary index file when there is not an existing one --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20402 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20373: [SPARK-23159][PYTHON] Update cloudpickle to match 0.4.2
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/20373 We should also backport https://github.com/cloudpipe/cloudpickle/pull/132? They say that it fixes the regression that affects 0.4.2 and later. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20303 **[Test build #86757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86757/testReport)** for PR 20303 at commit [`9a1301f`](https://github.com/apache/spark/commit/9a1301f1981a0725364b86c2aafe72a87110e7a3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20303 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86757/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20303: [SPARK-23128][SQL] A new approach to do adaptive executi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20303 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20397 **[Test build #86763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86763/testReport)** for PR 20397 at commit [`e570fca`](https://github.com/apache/spark/commit/e570fca9b16d320372869039305a12519e9b9b62). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86763/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20332: [SPARK-23138][ML][DOC] Multiclass logistic regres...
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/20332#discussion_r164387272 --- Diff: docs/ml-classification-regression.md --- @@ -125,7 +123,8 @@ Continuing the earlier example: [`LogisticRegressionTrainingSummary`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionSummary) provides a summary for a [`LogisticRegressionModel`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionModel). -Currently, only binary classification is supported. Support for multiclass model summaries will be added in the future. +In the case of binary classification, certain additional metrics are --- End diff -- Missing " The binary summary can be accessed via the ..." sentence in this one --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20350 kindly ping @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20375: [SPARK-23199][SQL]improved Removes repetition from group...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20375 **[Test build #86761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86761/testReport)** for PR 20375 at commit [`e1ff90e`](https://github.com/apache/spark/commit/e1ff90ed613372163cd93330b1d9e33f9873c6de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20375: [SPARK-23199][SQL]improved Removes repetition from group...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20375 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20375: [SPARK-23199][SQL]improved Removes repetition from group...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20375 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86761/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20373: [SPARK-23159][PYTHON] Update cloudpickle to match 0.4.2
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20373 Nice catch! Yup, I think we should do it too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20422 Is this just trying to reuse a file that should have been cleaned up after prior failure? If so is that possible as a more direct solution? I wonder if there aren't corner cases here where the file exists and it is still being written to by another process. This could result in corruption. But I am not familiar with this mechanism --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20403: [SPARK-23238][SQL] Externalize SQLConf configurations ex...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20403 Merged to master. Thanks @ueshin, @gatorsmile, @BryanCutler, @felixcheung and @viirya. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20403: [SPARK-23238][SQL] Externalize SQLConf configurat...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20403 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20422 I think it is necessary to add unit test to verify the changes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20408 **[Test build #86764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86764/testReport)** for PR 20408 at commit [`2c4c892`](https://github.com/apache/spark/commit/2c4c8927e2f56dbf09f47016a24d2daa4c5f6823). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFa...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20397#discussion_r164425194
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsScanColumnarBatch.java
---
@@ -30,21 +30,21 @@
@InterfaceStability.Evolving
public interface SupportsScanColumnarBatch extends DataSourceV2Reader {
@Override
- default List> createReadTasks() {
+ default List> createDataReaderFactories() {
--- End diff --
We mentioned it in the classdoc of `DataReaderFactory`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20408 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86764/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20408 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20397 **[Test build #86770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86770/testReport)** for PR 20397 at commit [`06739ca`](https://github.com/apache/spark/commit/06739ca2d5ef741dafb6a3024d72461a41657da6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20404: [SPARK-23228][PYSPARK] Add Python Created jsparkSession ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20404 **[Test build #86766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86766/testReport)** for PR 20404 at commit [`cc4b851`](https://github.com/apache/spark/commit/cc4b8510c1445fb742c0d750958d352adfa84902). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20421: [SPARK-23112][DOC] Update ML migration guide with breaki...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20421 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20402 **[Test build #86769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86769/testReport)** for PR 20402 at commit [`d0cfbd0`](https://github.com/apache/spark/commit/d0cfbd0b1975f0b35537eb2c573fe758048319f7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20421: [SPARK-23112][DOC] Update ML migration guide with breaki...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20421 **[Test build #86767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86767/testReport)** for PR 20421 at commit [`b6fcf15`](https://github.com/apache/spark/commit/b6fcf15dec51575bef0b1df787d57ba36b26ef63). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20421: [SPARK-23112][DOC] Update ML migration guide with breaki...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20421 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/333/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20422 **[Test build #86768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86768/testReport)** for PR 20422 at commit [`98ea6a7`](https://github.com/apache/spark/commit/98ea6a742143da803eb728c352e7424f504fabba). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/334/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFa...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20397#discussion_r164425827
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/DataReaderFactory.java
---
@@ -22,21 +22,23 @@
import org.apache.spark.annotation.InterfaceStability;
/**
- * A read task returned by {@link DataSourceV2Reader#createReadTasks()}
and is responsible for
- * creating the actual data reader. The relationship between {@link
ReadTask} and {@link DataReader}
+ * A reader factory returned by {@link
DataSourceV2Reader#createDataReaderFactories()} and is
+ * responsible for creating the actual data reader. The relationship
between
+ * {@link DataReaderFactory} and {@link DataReader}
* is similar to the relationship between {@link Iterable} and {@link
java.util.Iterator}.
*
- * Note that, the read task will be serialized and sent to executors, then
the data reader will be
- * created on executors and do the actual reading. So {@link ReadTask}
must be serializable and
- * {@link DataReader} doesn't need to be.
+ * Note that, the reader factory will be serialized and sent to executors,
then the data reader
+ * will be created on executors and do the actual reading. So {@link
DataReaderFactory} must be
+ * serializable and {@link DataReader} doesn't need to be.
*/
@InterfaceStability.Evolving
-public interface ReadTask extends Serializable {
+public interface DataReaderFactory extends Serializable {
/**
- * The preferred locations where this read task can run faster, but
Spark does not guarantee that
- * this task will always run on these locations. The implementations
should make sure that it can
- * be run on any location. The location is a string representing the
host name.
+ * The preferred locations where this data reader returned by this
reader factory can run faster,
--- End diff --
`this data reader` -> `the data reader`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20402: [SPARK-23223][SQL] Make stacking dataset transforms more...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20402 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/335/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFa...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20397#discussion_r164425992
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/DataReaderFactory.java
---
@@ -22,21 +22,23 @@
import org.apache.spark.annotation.InterfaceStability;
/**
- * A read task returned by {@link DataSourceV2Reader#createReadTasks()}
and is responsible for
- * creating the actual data reader. The relationship between {@link
ReadTask} and {@link DataReader}
+ * A reader factory returned by {@link
DataSourceV2Reader#createDataReaderFactories()} and is
+ * responsible for creating the actual data reader. The relationship
between
+ * {@link DataReaderFactory} and {@link DataReader}
* is similar to the relationship between {@link Iterable} and {@link
java.util.Iterator}.
*
- * Note that, the read task will be serialized and sent to executors, then
the data reader will be
- * created on executors and do the actual reading. So {@link ReadTask}
must be serializable and
- * {@link DataReader} doesn't need to be.
+ * Note that, the reader factory will be serialized and sent to executors,
then the data reader
+ * will be created on executors and do the actual reading. So {@link
DataReaderFactory} must be
+ * serializable and {@link DataReader} doesn't need to be.
*/
@InterfaceStability.Evolving
-public interface ReadTask extends Serializable {
+public interface DataReaderFactory extends Serializable {
/**
- * The preferred locations where this read task can run faster, but
Spark does not guarantee that
- * this task will always run on these locations. The implementations
should make sure that it can
- * be run on any location. The location is a string representing the
host name.
+ * The preferred locations where this data reader returned by this
reader factory can run faster,
+ * but Spark does not guarantee that this task will always run on these
locations.
--- End diff --
`not guarantee to always run the data reader on these locations.`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20397: [SPARK-23219][SQL]Rename ReadTask to DataReaderFactory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20397 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/336/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
