[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106421711 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -172,7 +172,7 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17316 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74667/ Test FAILed. ---

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17316 **[Test build #74667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74667/testReport)** for PR 17316 at commit

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17316 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17315: [SPARK-19949][SQL] unify bad record handling in C...

2017-03-16 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/17315 [SPARK-19949][SQL] unify bad record handling in CSV and JSON ## What changes were proposed in this pull request? Currently JSON and CSV have exactly the same logic about handling bad

[GitHub] spark issue #17292: [SPARK-19946][testing]DebugFilesystem.assertNoOpenStream...

2017-03-16 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/17292 LGTM - merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or wri...

2017-03-16 Thread windpiger
GitHub user windpiger opened a pull request: https://github.com/apache/spark/pull/17317 [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to a datasource table with a non pre-existing location should succeed ## What changes were proposed in this pull request? This is a

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17317 **[Test build #74671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74671/testReport)** for PR 17317 at commit

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-03-16 Thread aray
Github user aray commented on the issue: https://github.com/apache/spark/pull/16483 @rxin can anyone else review this? It would be nice to get this correctness fix into 2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17242: [SPARK-19902][SQL] Add optimization rule to simplify exp...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17242 **[Test build #74670 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74670/testReport)** for PR 17242 at commit

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17316 **[Test build #74669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74669/testReport)** for PR 17316 at commit

[GitHub] spark issue #17238: getRackForHost returns None if host is unknown by driver

2017-03-16 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17238 > Actually, to play devil's advocate, the problem @morenn520 is describing is a little more involved. You have a driver running, which has its own view of what the cluster topology is, and then

[GitHub] spark pull request #17291: [SPARK-19949][SQL][WIP] unify bad record handling...

2017-03-16 Thread cloud-fan
Github user cloud-fan closed the pull request at: https://github.com/apache/spark/pull/17291 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17316 **[Test build #74672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74672/testReport)** for PR 17316 at commit

[GitHub] spark pull request #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread MLnick
GitHub user MLnick opened a pull request: https://github.com/apache/spark/pull/17316 [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark Add Python wrapper for `Imputer` feature transformer. ## How was this patch tested? New doc tests and tweak to PySpark ML

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17316 cc @hhbyyh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74673/testReport)** for PR 16867 at commit

[GitHub] spark issue #16618: [SPARK-14409][ML][WIP] Add RankingEvaluator

2017-03-16 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/16618 The basic direction looks right - I won't have time to review immediately. Spark 2.2 QA code freeze will happen shortly so this will wait until 2.3 dev cycle starts --- If your project is set up

[GitHub] spark issue #17268: [SPARK-19932][SS] Disallow a case that might case OOM fo...

2017-03-16 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/17268 Sure; sorry I didn't say it out but I meant the same thing :-) @marmbrus now that I've updated this as well as the JIRA, would you mind taking another look? Thanks! --- If your project is

[GitHub] spark issue #17315: [SPARK-19949][SQL] unify bad record handling in CSV and ...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17315 **[Test build #74666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74666/testReport)** for PR 17315 at commit

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17317 cc @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #17292: [SPARK-19946][testing]DebugFilesystem.assertNoOpe...

2017-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17292 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...

2017-03-16 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/17274 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106433273 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -172,7 +172,7 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106434806 --- Diff: core/src/test/scala/org/apache/spark/util/collection/MedianHeapSuite.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106433116 --- Diff: core/src/test/scala/org/apache/spark/util/collection/MedianHeapSuite.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106436689 --- Diff: core/src/test/scala/org/apache/spark/util/collection/MedianHeapSuite.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17317 @windpiger Could you please close it? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16209: [WIP][SPARK-10849][SQL] Adds option to the JDBC data sou...

2017-03-16 Thread sureshthalamati
Github user sureshthalamati commented on the issue: https://github.com/apache/spark/pull/16209 @gatorsmile sure. I will update the PR with the DDL format approach. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/17316#discussion_r106488785 --- Diff: python/pyspark/ml/feature.py --- @@ -871,6 +872,164 @@ def idf(self): @inherit_doc +class Imputer(JavaEstimator, HasInputCols,

[GitHub] spark issue #17319: [SPARK-19765][SPARK-18549][SPARK-19093][SPARK-19736][BAC...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17319 **[Test build #74678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74678/testReport)** for PR 17319 at commit

[GitHub] spark issue #16905: [SPARK-19567][CORE][SCHEDULER] Support some Schedulable ...

2017-03-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16905 @shaneknapp I had to trigger jenkins manually via spark-prs. Every once in a while I encounter a pr for which tests are never triggered via comments. Its pretty rare, so its not a big deal, but I

[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...

2017-03-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17191 @gatorsmile I checked; ``` // PostgreSQL v9.5 postgres=# \d t Table "public.t" Column | Type | Modifiers +-+--- gkey | integer |

[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17274 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17295: [SPARK-19556][core] Do not encrypt block manager data in...

2017-03-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17295 > What's the actual difference? previously we transfer encrypted data? Yes. The previous version of the code would transfer the encrypted file over to the receiver, and the encrypted data

[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17191 **[Test build #74677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74677/testReport)** for PR 17191 at commit

[GitHub] spark pull request #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/17316#discussion_r106489373 --- Diff: python/pyspark/ml/feature.py --- @@ -871,6 +872,164 @@ def idf(self): @inherit_doc +class Imputer(JavaEstimator, HasInputCols,

[GitHub] spark pull request #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/17316#discussion_r106490851 --- Diff: python/pyspark/ml/feature.py --- @@ -871,6 +872,164 @@ def idf(self): @inherit_doc +class Imputer(JavaEstimator, HasInputCols,

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17316 **[Test build #74669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74669/testReport)** for PR 17316 at commit

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17316 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17316 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74669/ Test PASSed. ---

[GitHub] spark issue #17242: [SPARK-19902][SQL] Add optimization rule to simplify exp...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17242 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74670/ Test FAILed. ---

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74671/ Test PASSed. ---

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17317 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17274 **[Test build #74668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74668/testReport)** for PR 17274 at commit

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17318 @cloud-fan This is the same fix with https://github.com/apache/spark/pull/17188 and could you check this? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #17306: [SPARK-19979][MLLIB] Allow multiple pipelines when tunin...

2017-03-16 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17306 I commented on the linked JIRA also. In principle I think this can be a useful enhancement and yes the better efficiency on the caching side is a good benefit. I'd actually been thinking

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74674/ Test PASSed. ---

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74675/ Test PASSed. ---

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17287: [SPARK-19945][SQL]add test suite for SessionCatalog with...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17287 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #17287: [SPARK-19945][SQL]add test suite for SessionCatalog with...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17287 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74673/testReport)** for PR 16867 at commit

[GitHub] spark issue #17230: [SPARK-19353][CORE] Generalize PipedRDD to use I/O forma...

2017-03-16 Thread superbobry
Github user superbobry commented on the issue: https://github.com/apache/spark/pull/17230 Here're the result of piping [The Sonnets by W. Sheakespear](https://www.gutenberg.org/cache/epub/1105/pg1105.txt) via `cat` using either `PipedRDD` or hand-patched `PipedRDD` with I/O formats.

[GitHub] spark issue #16905: [SPARK-19567][CORE][SCHEDULER] Support some Schedulable ...

2017-03-16 Thread erenavsarogullari
Github user erenavsarogullari commented on the issue: https://github.com/apache/spark/pull/16905 Thanks again @kayousterhout and @squito ;) Jenkins needs to be retriggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74676/ Test PASSed. ---

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17318 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74674/testReport)** for PR 16867 at commit

[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17274 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74668/ Test PASSed. ---

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74673/ Test PASSed. ---

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17317 Thanks! Merging to 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17268: [SPARK-19932][SS] Disallow a case that might cause OOM f...

2017-03-16 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/17268 Sorry I'm still not sure if this is a good idea. Why disallow the following, ```scala spark .readStream .withWatermark("eventTime", "10 seconds")

[GitHub] spark issue #16905: [SPARK-19567][CORE][SCHEDULER] Support some Schedulable ...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16905 **[Test build #3600 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3600/testReport)** for PR 16905 at commit

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17318 **[Test build #74676 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74676/testReport)** for PR 17318 at commit

[GitHub] spark pull request #17319: [SPARK-19765][SPARK-18549][SPARK-19093][SPARK-197...

2017-03-16 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/17319 [SPARK-19765][SPARK-18549][SPARK-19093][SPARK-19736][BACKPORT-2.1][SQL] Backport Three Cache-related PRs to Spark 2.1 ### What changes were proposed in this pull request? Backport a

[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17191 We hit this issue multiple times. Although it looks not right to support it for the users of major enterprise RDBMS, two popular open source RDBMS PostgreSQL and MySQL support it. This

[GitHub] spark issue #17166: [SPARK-19820] [core] Allow reason to be specified for ta...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17166 **[Test build #74679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74679/testReport)** for PR 17166 at commit

[GitHub] spark issue #17166: [SPARK-19820] [core] Allow reason to be specified for ta...

2017-03-16 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/17166 Rebased --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74675/testReport)** for PR 16867 at commit

[GitHub] spark pull request #17287: [SPARK-19945][SQL]add test suite for SessionCatal...

2017-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17287 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17316 **[Test build #74672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74672/testReport)** for PR 17316 at commit

[GitHub] spark issue #17316: [SPARK-15040][ML][PYSPARK] Add Imputer to PySpark

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17316 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74672/ Test PASSed. ---

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17318 **[Test build #74676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74676/testReport)** for PR 17318 at commit

[GitHub] spark issue #17315: [SPARK-19949][SQL] unify bad record handling in CSV and ...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17315 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74666/ Test FAILed. ---

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106465419 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -175,6 +178,87 @@ case class AlterTableRenameCommand(

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106467818 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -175,6 +178,87 @@ case class AlterTableRenameCommand(

[GitHub] spark pull request #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFi...

2017-03-16 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/17274#discussion_r106471026 --- Diff: R/pkg/inst/tests/testthat/test_context.R --- @@ -177,6 +177,13 @@ test_that("add and get file to be downloaded with Spark job on every node",

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][WIP] R should check for java when ...

2017-03-16 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16596 @shivaram the change would need to be in spark-submit & spark-submit.cmd - that's where JAVA is resolved (and if it's not there, to empty string) --- If your project is set up for it, you can

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74674/testReport)** for PR 16867 at commit

[GitHub] spark issue #17276: [WIP][SPARK-19937] Collect metrics of block sizes when s...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17276 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74665/ Test PASSed. ---

[GitHub] spark issue #17276: [WIP][SPARK-19937] Collect metrics of block sizes when s...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17276 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17318: [SPARK-19896][SQL] Throw an exception if case cla...

2017-03-16 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/17318 [SPARK-19896][SQL] Throw an exception if case classes have circular references in toDS ## What changes were proposed in this pull request? If case classes have circular references below, it

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #74675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74675/testReport)** for PR 16867 at commit

[GitHub] spark issue #17315: [SPARK-19949][SQL] unify bad record handling in CSV and ...

2017-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17315 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17315: [SPARK-19949][SQL] unify bad record handling in CSV and ...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17315 **[Test build #74666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74666/testReport)** for PR 17315 at commit

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106470033 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -87,8 +87,13 @@ case class

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106468328 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerSource.scala --- @@ -26,35 +26,39 @@ private[spark] class BlockManagerSource(val

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106467740 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage.js --- @@ -350,7 +366,12 @@ $(document).ready(function () {

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106467583 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage.js --- @@ -174,8 +174,10 @@ $(document).ready(function () {

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106467285 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html --- @@ -24,7 +24,10 @@ Summary RDD

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106467368 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html --- @@ -24,7 +24,10 @@ Summary RDD

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106469513 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala --- @@ -116,7 +121,11 @@ private[spark] object ExecutorsPage {

[GitHub] spark pull request #14617: [SPARK-17019][Core] Expose on-heap and off-heap m...

2017-03-16 Thread jsoltren
Github user jsoltren commented on a diff in the pull request: https://github.com/apache/spark/pull/14617#discussion_r106469348 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -60,11 +63,17 @@ class StorageStatus(val blockManagerId: BlockManagerId,

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106472336 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -175,6 +178,87 @@ case class AlterTableRenameCommand(

[GitHub] spark issue #17317: [SPARK-19329][SQL][BRANCH-2.1]Reading from or writing to...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17317 **[Test build #74671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74671/testReport)** for PR 17317 at commit

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-16 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106472274 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -175,6 +178,87 @@ case class AlterTableRenameCommand(

[GitHub] spark issue #17242: [SPARK-19902][SQL] Add optimization rule to simplify exp...

2017-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17242 **[Test build #74670 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74670/testReport)** for PR 17242 at commit

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-03-16 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r106453502 --- Diff: core/src/test/scala/org/apache/spark/util/collection/MedianHeapSuite.scala --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software

  1   2   3   4   5   6   >