[GitHub] spark pull request #21092: [SPARK-23984][K8S] Initial Python Bindings for Py...

2018-06-06 Thread kokes
Github user kokes commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r193639554 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -154,6 +176,24 @@ private[spark] object Config ext

[GitHub] spark issue #20929: [SPARK-23772][SQL] Provide an option to ignore column of...

2018-06-06 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20929 yea, thanks for the comments! I'll try to fix based on the comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apach

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193290266 --- Diff: python/pyspark/worker.py --- @@ -232,6 +236,13 @@ def main(infile, outfile): shuffle.DiskBytesSpilled = 0 _accumulatorRegist

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193269255 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkCo

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193289530 --- Diff: python/pyspark/worker.py --- @@ -232,6 +236,13 @@ def main(infile, outfile): shuffle.DiskBytesSpilled = 0 _accumulatorRegist

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193291076 --- Diff: python/pyspark/worker.py --- @@ -232,6 +236,13 @@ def main(infile, outfile): shuffle.DiskBytesSpilled = 0 _accumulatorRegist

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193555968 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -123,6 +124,21 @@ private[spark] class TaskSetManager( // TODO: We

[GitHub] spark pull request #21494: [WIP][SPARK-24375][Prototype] Support barrier sch...

2018-06-06 Thread galv
Github user galv commented on a diff in the pull request: https://github.com/apache/spark/pull/21494#discussion_r193269297 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkCo

[GitHub] spark issue #21500: Scalable Memory option for HDFSBackedStateStore

2018-06-06 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21500 Retaining versions of state is also relevant to do snapshotting the last version in files: HDFSBackedStateStoreProvider doesn't snapshot if the version doesn't exist in loadedMaps. So we may wan

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r193635361 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala --- @@ -84,7 +86,36 @@ class StopWordsRemover @Since("1.5.0") (@Since("1.5.

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193634436 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): sel

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193633540 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): sel

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193632947 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): self._jwri

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193632883 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): sel

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91516/testReport)** for PR 21483 at commit [`49323a6`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91516/ Test PASSed. ---

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193631566 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): self._jwri

[GitHub] spark issue #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21482 **[Test build #91518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91518/testReport)** for PR 21482 at commit [`6a4d46e`](https://github.com/apache/spark/commit/6a

[GitHub] spark pull request #21477: [WIP] [SPARK-24396] [SS] [PYSPARK] Add Structured...

2018-06-06 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/21477#discussion_r193631209 --- Diff: python/pyspark/sql/streaming.py --- @@ -843,6 +844,169 @@ def trigger(self, processingTime=None, once=None, continuous=None): self._jwri

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91515/testReport)** for PR 21483 at commit [`5e293d5`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91515/ Test PASSed. ---

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3829/ Tes

[GitHub] spark pull request #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21482#discussion_r193630041 --- Diff: R/pkg/NAMESPACE --- @@ -281,6 +281,8 @@ exportMethods("%<=>%", "initcap", "input_file_name",

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91516/testReport)** for PR 21483 at commit [`49323a6`](https://github.com/apache/spark/commit/49

[GitHub] spark issue #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21482 **[Test build #91517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91517/testReport)** for PR 21482 at commit [`f240fdf`](https://github.com/apache/spark/commit/f2

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91514/ Test PASSed. ---

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports submodule...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91514/testReport)** for PR 21483 at commit [`55eef7c`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3828/ Tes

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91515/testReport)** for PR 21483 at commit [`5e293d5`](https://github.com/apache/spark/commit/5e

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21483 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3827/ Tes

[GitHub] spark issue #21483: [SPARK-24477][SPARK-24454][ML][PYTHON] Imports image mod...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21483 **[Test build #91514 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91514/testReport)** for PR 21483 at commit [`55eef7c`](https://github.com/apache/spark/commit/55

[GitHub] spark issue #21504: SPARK-24479: Added config for registering streamingQuery...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91510/ Test PASSed. ---

[GitHub] spark issue #21504: SPARK-24479: Added config for registering streamingQuery...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21504: SPARK-24479: Added config for registering streamingQuery...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91510/testReport)** for PR 21504 at commit [`d3a3baa`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21500: Scalable Memory option for HDFSBackedStateStore

2018-06-06 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21500 @TomaszGaweda @aalobaidi Please correct me if I'm missing here. From every start of batch, state store loads previous version of state so that it can be read and written. If we unlo

[GitHub] spark pull request #21469: [SPARK-24441][SS] Expose total estimated size of ...

2018-06-06 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/21469#discussion_r193622940 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala --- @@ -231,7 +231,7 @@ class StreamingQueryListener

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91513/ Test FAILed. ---

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91513/testReport)** for PR 18900 at commit [`e3a0cc4`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91513/testReport)** for PR 18900 at commit [`e3a0cc4`](https://github.com/apache/spark/commit/e3

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91512/ Test FAILed. ---

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91512/testReport)** for PR 18900 at commit [`a00e943`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91512/testReport)** for PR 18900 at commit [`a00e943`](https://github.com/apache/spark/commit/a0

[GitHub] spark pull request #21499: [SPARK-24468][SQL] Handle negative scale when adj...

2018-06-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21499#discussion_r193618762 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala --- @@ -161,13 +161,17 @@ object DecimalType extends AbstractDataType {

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91509/ Test PASSed. ---

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21469 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21469 **[Test build #91509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91509/testReport)** for PR 21469 at commit [`7ec3242`](https://github.com/apache/spark/commit/7

[GitHub] spark pull request #21498: [SPARK-24410][SQL][Core] Optimization for Union o...

2018-06-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21498#discussion_r193618338 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1099,6 +1099,17 @@ object SQLConf { .intConf

[GitHub] spark issue #21498: [SPARK-24410][SQL][Core] Optimization for Union outputPa...

2018-06-06 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21498 I set up a Spark cluster with 5 nodes on EC2. ```scala def benchmark(func: () => Unit): Unit = { val t0 = System.nanoTime() func() val t1 = System.nanoTime()

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91511 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91511/testReport)** for PR 18900 at commit [`478e205`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91511/ Test FAILed. ---

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18900 **[Test build #91511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91511/testReport)** for PR 18900 at commit [`478e205`](https://github.com/apache/spark/commit/47

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18900 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spa

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18900 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #18900: [SPARK-21687][SQL] Spark SQL should set createTim...

2018-06-06 Thread debugger87
GitHub user debugger87 reopened a pull request: https://github.com/apache/spark/pull/18900 [SPARK-21687][SQL] Spark SQL should set createTime for Hive partition ## What changes were proposed in this pull request? Set createTime for every hive partition created in Spark SQL,

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

2018-06-06 Thread debugger87
Github user debugger87 commented on the issue: https://github.com/apache/spark/pull/18900 @cxzl25 OK, reopen it --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark pull request #21501: [SPARK-15064][ML] Locale support in StopWordsRemo...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21501#discussion_r193604620 --- Diff: python/pyspark/ml/feature.py --- @@ -2582,25 +2582,27 @@ class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol, JavaMLReadabl

[GitHub] spark pull request #21504: SPARK-24479: Added config for registering streami...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21504#discussion_r193604356 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenersConfSuite.scala --- @@ -0,0 +1,66 @@ +/* + * Licensed t

[GitHub] spark pull request #21504: SPARK-24479: Added config for registering streami...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21504#discussion_r193603810 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -55,6 +56,12 @@ class StreamingQueryManager private

[GitHub] spark issue #21504: SPARK-24479: Added config for registering streamingQuery...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21504 Mind fixing the PR title to `[SPARK-24479][SS] Added config for registering streamingQueryListeners`? --- - To unsubscribe,

[GitHub] spark issue #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-06 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/21482 @henryr 1.0/0.0 also returns null in Spark SQL ... ``` scala> sql("select cast(1.0 as double)/cast(0 as double)").show() +-+ |(CAST(1.0 AS DOU

[GitHub] spark issue #21504: SPARK-24480: Added config for registering streamingQuery...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21504 **[Test build #91510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91510/testReport)** for PR 21504 at commit [`d3a3baa`](https://github.com/apache/spark/commit/d3

[GitHub] spark pull request #21504: SPARK-24480: Added config for registering streami...

2018-06-06 Thread arunmahadevan
GitHub user arunmahadevan opened a pull request: https://github.com/apache/spark/pull/21504 SPARK-24480: Added config for registering streamingQueryListeners ## What changes were proposed in this pull request? Currently a "StreamingQueryListener" can only be registered prog

[GitHub] spark issue #21504: SPARK-24480: Added config for registering streamingQuery...

2018-06-06 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21504 ping @tdas @jose-torres @HeartSaVioR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional com

[GitHub] spark issue #21504: SPARK-24480: Added config for registering streamingQuery...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21504 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21503 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91507/ Test PASSed. ---

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21503 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21503 **[Test build #91507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91507/testReport)** for PR 21503 at commit [`9d3a11e`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21469 **[Test build #91509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91509/testReport)** for PR 21469 at commit [`7ec3242`](https://github.com/apache/spark/commit/7e

[GitHub] spark issue #21482: [SPARK-24393][SQL] SQL builtin: isinf

2018-06-06 Thread henryr
Github user henryr commented on the issue: https://github.com/apache/spark/pull/21482 @rxin Other engines are all over the place: * MySQL doesn't have support for infinity (based on my cursory look) - 1.0 / 0.0 is written as `null`. Also seems to be true of SQLite. * Postg

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 There is only target/surefire-reports/TEST-org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite.xml under target/surefire-reports That file doesn't contain test output. --- -

[GitHub] spark issue #20929: [SPARK-23772][SQL] Provide an option to ignore column of...

2018-06-06 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/20929 @maropu Thanks for updating this PR! It would be easier to maintain the logic in one place. I think it should be feasible to do everything inside `canonicalizeType` without modifying `JsonParser` or

[GitHub] spark pull request #21488: SPARK-18057 Update structured streaming kafka fro...

2018-06-06 Thread guozhangwang
Github user guozhangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21488#discussion_r193574097 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala --- @@ -203,7 +215,13 @@ class KafkaTestUtils(with

[GitHub] spark issue #21194: [SPARK-24046][SS] Fix rate source when rowsPerSecond <= ...

2018-06-06 Thread maasg
Github user maasg commented on the issue: https://github.com/apache/spark/pull/21194 @zsxwing Thanks for dropping by. This patch is about fixing the rate ramp up when `rowsPerSecond <= rampUpTime`, which makes the Rate Source produce no data until `rampUpTime` (See [SPARK-24046](htt

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3826/ Tes

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #91508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91508/testReport)** for PR 21488 at commit [`b773982`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91508/ Test FAILed. ---

[GitHub] spark issue #21488: SPARK-18057 Update structured streaming kafka from 0.10....

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #91508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91508/testReport)** for PR 21488 at commit [`b773982`](https://github.com/apache/spark/commit/b7

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21503 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3825/ Tes

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21503 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21488: SPARK-18057 Update structured streaming kafka fro...

2018-06-06 Thread eric-maynard
Github user eric-maynard commented on a diff in the pull request: https://github.com/apache/spark/pull/21488#discussion_r193549547 --- Diff: external/kafka-0-10-sql/pom.xml --- @@ -29,7 +29,7 @@ spark-sql-kafka-0-10_2.11 sql-kafka-0-10 -0.10.0.1 +

[GitHub] spark issue #21319: [SPARK-24267][SQL] explicitly keep DataSourceReader in D...

2018-06-06 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21319 Here's the commit with my changes to support v2 stats in the visitor, sorry it took so long for me to find the time! https://github.com/apache/spark/pull/21503/commits/9d3a11e68bca6c5a56a2be47fb0

[GitHub] spark issue #21503: [SPARK-24478][SQL] Move projection and filter push down ...

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21503 **[Test build #91507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91507/testReport)** for PR 21503 at commit [`9d3a11e`](https://github.com/apache/spark/commit/9d

[GitHub] spark pull request #21503: [SPARK-24478][SQL] Move projection and filter pus...

2018-06-06 Thread rdblue
GitHub user rdblue opened a pull request: https://github.com/apache/spark/pull/21503 [SPARK-24478][SQL] Move projection and filter push down to physical conversion ## What changes were proposed in this pull request? This removes the v2 optimizer rule for push-down and inste

[GitHub] spark issue #21180: [SPARK-22674][PYTHON] Disabled _hack_namedtuple for pick...

2018-06-06 Thread superbobry
Github user superbobry commented on the issue: https://github.com/apache/spark/pull/21180 Friendly ping @HyukjinKwon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21502: [SPARK-22575][SQL] Add destroy to Dataset

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21502 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91505/ Test FAILed. ---

[GitHub] spark issue #21502: [SPARK-22575][SQL] Add destroy to Dataset

2018-06-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21502 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21502: [SPARK-22575][SQL] Add destroy to Dataset

2018-06-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21502 **[Test build #91505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91505/testReport)** for PR 21502 at commit [`ec365d6`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21483: [SPARK-24454][ML][PYTHON] Imports image module in ml/__i...

2018-06-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21483 Ah, maybe I rushed to read the JIRA. For 2. there are already single attribute being loaded `ImageSchema` since all other attributes has an underscore; however, sure, it should be the best to ex

  1   2   >