[GitHub] spark issue #17172: [SPARK-19008][SQL] Improve performance of Dataset.map by...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17172 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17172: [SPARK-19008][SQL] Improve performance of Dataset...

2017-03-08 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/17172#discussion_r105103962 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala --- @@ -219,7 +219,33 @@ case class MapElementsExec( override def

[GitHub] spark issue #17172: [SPARK-19008][SQL] Improve performance of Dataset.map by...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17172 **[Test build #74249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74249/testReport)** for PR 17172 at commit

[GitHub] spark pull request #17172: [SPARK-19008][SQL] Improve performance of Dataset...

2017-03-08 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/17172#discussion_r105103907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala --- @@ -219,7 +219,33 @@ case class MapElementsExec( override def

[GitHub] spark pull request #17175: [SPARK-19468][SQL] Rewrite physical Project opera...

2017-03-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17175#discussion_r105103474 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -78,9 +78,42 @@ case class ProjectExec(projectList:

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16729 One other change I could make is to change `variancePower` and `linkPower` to `var.power` and `link.power` to be consistent with `statmod`. But l would like to get your feedback on this new

[GitHub] spark issue #16541: [SPARK-19088][SQL] Optimize sequence type deserializatio...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16541 is it a performance improvement? there is no difference in the benchmark results --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16729 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74243/ Test PASSed. ---

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16729 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16729 **[Test build #74243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74243/testReport)** for PR 16729 at commit

[GitHub] spark pull request #17175: [SPARK-19468][SQL] Rewrite physical Project opera...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17175#discussion_r105102645 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -78,9 +78,42 @@ case class ProjectExec(projectList:

[GitHub] spark pull request #17175: [SPARK-19468][SQL] Rewrite physical Project opera...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17175#discussion_r105102603 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -78,9 +78,42 @@ case class ProjectExec(projectList:

[GitHub] spark pull request #17223: [SPARK-19881][SQL] Support Dynamic Partition Inse...

2017-03-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/17223#discussion_r105102457 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -793,6 +794,20 @@ private[spark] class

[GitHub] spark pull request #17175: [SPARK-19468][SQL] Rewrite physical Project opera...

2017-03-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17175#discussion_r105102081 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Canonicalize.scala --- @@ -43,6 +43,12 @@ object Canonicalize extends {

[GitHub] spark pull request #17175: [SPARK-19468][SQL] Rewrite physical Project opera...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17175#discussion_r105101933 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Canonicalize.scala --- @@ -43,6 +43,12 @@ object Canonicalize extends {

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-08 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r105101799 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -1339,6 +1339,11 @@ test_that("column functions", { expect_equal(collect(select(df,

[GitHub] spark issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint prop...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17186 **[Test build #74248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74248/testReport)** for PR 17186 at commit

[GitHub] spark issue #17223: [SPARK-19881][SQL] Support Dynamic Partition Inserts par...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17223 **[Test build #74247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74247/testReport)** for PR 17223 at commit

[GitHub] spark pull request #17223: [SPARK-19881][SQL] Support Dynamic Partition Inse...

2017-03-08 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/17223 [SPARK-19881][SQL] Support Dynamic Partition Inserts params with SET command ## What changes were proposed in this pull request? Since Spark 2.0.0, `SET` commands do not pass the

[GitHub] spark pull request #17221: [SPARK-19859][SS][Follow-up] The new watermark sh...

2017-03-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17221 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFuncti...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17222 **[Test build #74246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74246/testReport)** for PR 17222 at commit

[GitHub] spark issue #17221: [SPARK-19859][SS][Follow-up] The new watermark should ov...

2017-03-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17221 Thanks! LGTM. Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJav...

2017-03-08 Thread zjffdu
GitHub user zjffdu opened a pull request: https://github.com/apache/spark/pull/17222 [SPARK-19439][PYSPARK][SQL] PySpark's registerJavaFunction Should Support UDAFs ## What changes were proposed in this pull request? Support register Java UDAFs in PySpark so that user can

[GitHub] spark pull request #17217: [SPARK-19874][BUILD]Hide API docs for org.apache....

2017-03-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17217 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15628: [SPARK-17471][ML] Add compressed method to ML mat...

2017-03-08 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/15628#discussion_r105032243 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -153,6 +153,86 @@ sealed trait Matrix extends Serializable {

[GitHub] spark pull request #15628: [SPARK-17471][ML] Add compressed method to ML mat...

2017-03-08 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/15628#discussion_r105068149 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -153,6 +153,86 @@ sealed trait Matrix extends Serializable {

[GitHub] spark pull request #15628: [SPARK-17471][ML] Add compressed method to ML mat...

2017-03-08 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/15628#discussion_r87668326 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -153,6 +153,86 @@ sealed trait Matrix extends Serializable { */

[GitHub] spark pull request #15628: [SPARK-17471][ML] Add compressed method to ML mat...

2017-03-08 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/15628#discussion_r87669463 --- Diff: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala --- @@ -153,6 +153,86 @@ sealed trait Matrix extends Serializable { */

[GitHub] spark issue #17217: [SPARK-19874][BUILD]Hide API docs for org.apache.spark.s...

2017-03-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17217 Thanks! Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16592: [SPARK-19235] [SQL] [TESTS] Enable Test Cases in ...

2017-03-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16592 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16592: [SPARK-19235] [SQL] [TESTS] Enable Test Cases in DDLSuit...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16592 great work! merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16592: [SPARK-19235] [SQL] [TESTS] Enable Test Cases in ...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16592#discussion_r105098789 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala --- @@ -306,6 +306,11 @@ private[sql] trait SQLTestUtils val fs =

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17120 **[Test build #74245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74245/testReport)** for PR 17120 at commit

[GitHub] spark pull request #16592: [SPARK-19235] [SQL] [TESTS] Enable Test Cases in ...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16592#discussion_r105097838 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -115,6 +229,11 @@ class DDLSuite extends QueryTest with

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r105097834 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -1339,6 +1339,11 @@ test_that("column functions", { expect_equal(collect(select(df,

[GitHub] spark issue #17194: Add new aggregates EVERY and ANY (SOME).

2017-03-08 Thread zjffdu
Github user zjffdu commented on the issue: https://github.com/apache/spark/pull/17194 @ptkool Please help the title to include the JIRA Id so that it can be linked to jira automatically. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request #17120: [SPARK-19715][Structured Streaming] Option to Str...

2017-03-08 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/17120#discussion_r105097597 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala --- @@ -79,9 +81,16 @@ class FileStreamSource(

[GitHub] spark pull request #17174: [SPARK-19145][SQL] Timestamp to String casting is...

2017-03-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17174#discussion_r105097503 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -324,14 +324,22 @@ object TypeCoercion {

[GitHub] spark issue #17200: [SPARK-19561][SQL] add int case handling for TimestampTy...

2017-03-08 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17200 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16729 **[Test build #74243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74243/testReport)** for PR 16729 at commit

[GitHub] spark issue #15363: [SPARK-17791][SQL] Join reordering using star schema det...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15363 **[Test build #74244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74244/testReport)** for PR 15363 at commit

[GitHub] spark issue #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17170 Sure - it's a single columns called `features` so I'm fine with it as a parameter. I'm not sure about `inputCol` though, it's a different nomenclature in R that is different from the rest

[GitHub] spark issue #16933: [SPARK-19601] [SQL] Fix CollapseRepartition rule to pres...

2017-03-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16933 awesome! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2017-03-08 Thread zdove
Github user zdove commented on the issue: https://github.com/apache/spark/pull/14239 Thank you for your reply. I consider that the code you have changed don't involve about SparkConf. If you don't change the default SparkConf file then Spark will not recognize the parameter

[GitHub] spark issue #17200: [SPARK-19561][SQL] add int case handling for TimestampTy...

2017-03-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17200 oh, the PR description is not correct now. Can you update it too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16729 **[Test build #74242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74242/testReport)** for PR 16729 at commit

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16729 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74242/ Test FAILed. ---

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16729 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17178: [SPARK-19828][R] Support array type in from_json ...

2017-03-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17178#discussion_r105095002 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -1342,28 +1342,52 @@ test_that("column functions", { df <- read.json(mapTypeJsonPath)

[GitHub] spark pull request #16909: [SPARK-13450] Introduce ExternalAppendOnlyUnsafeR...

2017-03-08 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/16909#discussion_r105094904 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -674,6 +675,24 @@ object SQLConf { .stringConf

[GitHub] spark pull request #17120: [SPARK-19715][Structured Streaming] Option to Str...

2017-03-08 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17120#discussion_r105094841 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala --- @@ -79,9 +81,16 @@ class FileStreamSource(

[GitHub] spark issue #17200: [SPARK-19561][SQL] cast TimestampType.toInternal output ...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17200 **[Test build #74240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74240/testReport)** for PR 17200 at commit

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16729 **[Test build #74242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74242/testReport)** for PR 16729 at commit

[GitHub] spark issue #16909: [SPARK-13450] Introduce ExternalAppendOnlyUnsafeRowArray...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16909 **[Test build #74241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74241/testReport)** for PR 16909 at commit

[GitHub] spark issue #16729: [SPARK-19391][SparkR][ML] Tweedie GLM API for SparkR

2017-03-08 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16729 @felixcheung OK, new implementation of # 3. Now works in two ways: 1. `family = "tweedie"` + `variancePower` + `linkPower` 2. When `statmod` is available, `tweedie()` Please

[GitHub] spark issue #17200: [SPARK-19561][SQL] cast TimestampType.toInternal output ...

2017-03-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17200 LGTM except for a minor comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16909: [SPARK-13450] Introduce ExternalAppendOnlyUnsafeR...

2017-03-08 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/16909#discussion_r105093717 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala --- @@ -164,9 +176,12 @@ private[window] final class

[GitHub] spark pull request #16909: [SPARK-13450] Introduce ExternalAppendOnlyUnsafeR...

2017-03-08 Thread tejasapatil
Github user tejasapatil commented on a diff in the pull request: https://github.com/apache/spark/pull/16909#discussion_r105093397 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala --- @@ -341,25 +364,27 @@ private[window] final

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2017-03-08 Thread f7753
Github user f7753 commented on the issue: https://github.com/apache/spark/pull/14239 @tgravescs Sorry for so late to reply, I am really sorry for that. I haven't use external shuffle, I need to rebase the code to recent release and retest it. I‘ll rebase it on executor

[GitHub] spark issue #17200: [SPARK-19561][Python] cast TimestampType.toInternal outp...

2017-03-08 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/17200 Btw, since it is a change to SQL code. Better to add `[SQL]` to the title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17207: [SPARK-19282][ML][SparkR] Expose model param "maxDepth" ...

2017-03-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17207 ok, great, could you update PR title and JIRA title then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17200: [SPARK-19561][Python] cast TimestampType.toIntern...

2017-03-08 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17200#discussion_r105094257 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala --- @@ -112,6 +112,7 @@ object EvaluatePython { case

[GitHub] spark issue #17200: [SPARK-19561][Python] cast TimestampType.toInternal outp...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17200 **[Test build #74239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74239/testReport)** for PR 17200 at commit

[GitHub] spark issue #17200: [SPARK-19561][Python] cast TimestampType.toInternal outp...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17200 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74239/ Test FAILed. ---

[GitHub] spark issue #17200: [SPARK-19561][Python] cast TimestampType.toInternal outp...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17200 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17200: [SPARK-19561][Python] cast TimestampType.toInternal outp...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17200 **[Test build #74239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74239/testReport)** for PR 17200 at commit

[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17141 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17141 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74238/ Test PASSed. ---

[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17141 **[Test build #74238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74238/testReport)** for PR 17141 at commit

[GitHub] spark issue #17179: [SPARK-19067][SS] Processing-time-based timeout in MapGr...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17179 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17179: [SPARK-19067][SS] Processing-time-based timeout in MapGr...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17179 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74235/ Test PASSed. ---

[GitHub] spark issue #17179: [SPARK-19067][SS] Processing-time-based timeout in MapGr...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17179 **[Test build #74235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74235/testReport)** for PR 17179 at commit

[GitHub] spark issue #17221: [SPARK-19859][SS][Follow-up] The new watermark should ov...

2017-03-08 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17221 cc @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #14239: [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism...

2017-03-08 Thread f7753
Github user f7753 commented on the issue: https://github.com/apache/spark/pull/14239 @zdove please refer to `org.apache.spark.SparkConf`, spark load all parameter you set on the file spark-defaults by scripts when JVM process start and filter those contains "spark", then add them to

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-08 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r105089766 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,340 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-08 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r105089751 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -389,6 +389,18 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-08 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r105089699 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -167,7 +167,8 @@ object

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-08 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r105089676 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -257,3 +258,28 @@ object PhysicalAggregation {

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17120 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74236/ Test PASSed. ---

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17120 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17120 **[Test build #74236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74236/testReport)** for PR 17120 at commit

[GitHub] spark issue #17113: [SPARK-13669][Core] Improve the blacklist mechanism to h...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17113 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74234/ Test PASSed. ---

[GitHub] spark issue #17113: [SPARK-13669][Core] Improve the blacklist mechanism to h...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17113 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17113: [SPARK-13669][Core] Improve the blacklist mechanism to h...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17113 **[Test build #74234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74234/testReport)** for PR 17113 at commit

[GitHub] spark issue #15229: [SPARK-17654] [SQL] Propagate bucketing information for ...

2017-03-08 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/15229 @carlos-verdes : Thanks for the information. This is moved under an umbrella jira (SPARK-19256) which has a proposal :

[GitHub] spark pull request #15229: [SPARK-17654] [SQL] Propagate bucketing informati...

2017-03-08 Thread tejasapatil
Github user tejasapatil closed the pull request at: https://github.com/apache/spark/pull/15229 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17138: [SPARK-17080] [SQL] join reorder

2017-03-08 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17138 CBO should be part of planning, right? The result of CBO should be executable plans. IMO, all the CBO rules should be moved the planner. The current optimizer is just rule-based.

[GitHub] spark issue #17221: [SPARK-19859][SS][Follow-up] The new watermark should ov...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17221 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74233/ Test PASSed. ---

[GitHub] spark issue #17221: [SPARK-19859][SS][Follow-up] The new watermark should ov...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17221 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17221: [SPARK-19859][SS][Follow-up] The new watermark should ov...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17221 **[Test build #74233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74233/testReport)** for PR 17221 at commit

[GitHub] spark pull request #17200: [SPARK-19561][Python] cast TimestampType.toIntern...

2017-03-08 Thread JasonMWhite
Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/17200#discussion_r105083494 --- Diff: python/pyspark/sql/types.py --- @@ -189,7 +189,7 @@ def toInternal(self, dt): if dt is not None: seconds =

[GitHub] spark issue #17209: [SPARK-19853][SS] uppercase kafka topics fail when start...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74237/ Test PASSed. ---

[GitHub] spark issue #17209: [SPARK-19853][SS] uppercase kafka topics fail when start...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17209 **[Test build #74237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74237/testReport)** for PR 17209 at commit

[GitHub] spark issue #17209: [SPARK-19853][SS] uppercase kafka topics fail when start...

2017-03-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17209 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17209: [SPARK-19853][SS] uppercase kafka topics fail when start...

2017-03-08 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17209 cc @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #17189: [SPARK-19831][CORE] Use a separate thread to clea...

2017-03-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/17189#discussion_r105081718 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -577,13 +582,17 @@ private[deploy] class Worker( val

[GitHub] spark issue #17209: [SPARK-19853][SS] uppercase kafka topics fail when start...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17209 **[Test build #74237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74237/testReport)** for PR 17209 at commit

[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17141 **[Test build #74238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74238/testReport)** for PR 17141 at commit

[GitHub] spark pull request #17216: [SPARK-19873][SS] Record num shuffle partitions i...

2017-03-08 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/17216#discussion_r105081023 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala --- @@ -71,7 +71,10 @@ object OffsetSeq { * @param

[GitHub] spark issue #17120: [SPARK-19715][Structured Streaming] Option to Strip Path...

2017-03-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17120 **[Test build #74236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74236/testReport)** for PR 17120 at commit

  1   2   3   4   5   6   >