[GitHub] spark pull request #17330: [SPARK-19993][SQL] Caching logical plans containi...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17330#discussion_r106795140 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -61,6 +63,37 @@ abstract class

[GitHub] spark pull request #17330: [SPARK-19993][SQL] Caching logical plans containi...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17330#discussion_r106795081 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -83,6 +116,19 @@ object SubqueryExpression {

[GitHub] spark issue #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17170 Could you update this PR to have the parameter itemsCol And remove predictionCol (if I recall, we don't expose that in the R API for other models either) --- If your project is

[GitHub] spark pull request #17330: [SPARK-19993][SQL] Caching logical plans containi...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17330#discussion_r106795011 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -83,6 +116,19 @@ object SubqueryExpression {

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74800/testReport)** for PR 17343 at commit

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16971 LGTM pending Jenkins cc @thunterdb @MLnick --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74801/testReport)** for PR 16596 at commit

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 cc - @rxin, @squito, @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74799/testReport)** for PR 16596 at commit

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74798/testReport)** for PR 17343 at commit

[GitHub] spark pull request #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream ...

2017-03-18 Thread sitalkedia
GitHub user sitalkedia opened a pull request: https://github.com/apache/spark/pull/17343 [SPARK-20014] Optimize mergeSpillsWithFileStream method ## What changes were proposed in this pull request? When the individual partition size in a spill is small,

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74797/testReport)** for PR 16596 at commit

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74796/testReport)** for PR 16596 at commit

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16596 updated. pretty sure this is an issue on Windows only --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74795/testReport)** for PR 16596 at commit

[GitHub] spark issue #17341: [SPARK-20013][SQL]add a newTablePath parameter for renam...

2017-03-18 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17341 cc @cloud-fan @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...

2017-03-18 Thread windpiger
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17338 yes,it is. It (jar:file://) will be re-resolved by new Path later, and will throw an exception described in the jira. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #74794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74794/testReport)** for PR 16330 at commit

[GitHub] spark pull request #16330: [SPARK-18817][SPARKR][SQL] change derby log outpu...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16330#discussion_r106794245 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -2909,6 +2910,30 @@ test_that("Collect on DataFrame when NAs exists at the top of a

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r106793811 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -425,8 +425,8 @@ object FunctionRegistry {

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16971 **[Test build #74793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74793/testReport)** for PR 16971 at commit

[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16971 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r106794013 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -624,41 +627,58 @@ case class

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106794032 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,340 @@ package

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r106793959 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -624,41 +627,58 @@ case class

[GitHub] spark pull request #17192: [SPARK-19849][SQL] Support ArrayType in to_json t...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17192#discussion_r106793800 --- Diff: python/pyspark/sql/functions.py --- @@ -1774,10 +1774,11 @@ def json_tuple(col, *fields): def from_json(col, schema, options={}):

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106793947 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/StarJoinSuite.scala --- @@ -0,0 +1,488 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106793898 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #16982: [SPARK-19654][SPARKR][SS] Structured Streaming AP...

2017-03-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16982 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74791/ Test PASSed. ---

[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16626 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16626 **[Test build #74791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74791/testReport)** for PR 16626 at commit

[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...

2017-03-18 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/17250 @budde Do you think you can update this PR? The 2.2 branch will be cut on Monday (2017-03-18). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16626 Adding a check in the existing test case to see if `HIVE_TYPE_STRING` is correctly populated in the metadata. LGTM except a few minor comments cc @cloud-fan --- If your

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106792988 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -2178,4 +2177,136 @@ abstract class DDLSuite extends

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106792941 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -175,6 +178,74 @@ case class AlterTableRenameCommand(

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106792952 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala --- @@ -165,7 +165,6 @@ class InMemoryCatalogedDDLSuite extends

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106792918 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala --- @@ -450,6 +451,21 @@ abstract class

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106792896 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala --- @@ -450,6 +451,21 @@ abstract class

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #74792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74792/testReport)** for PR 17342 at commit

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread weiqingy
Github user weiqingy commented on the issue: https://github.com/apache/spark/pull/17342 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread weiqingy
Github user weiqingy commented on the issue: https://github.com/apache/spark/pull/17342 `org.apache.spark.storage.BlockManagerProactiveReplicationSuite.proactive block replication - 3 replicas - 2 block manager deletions` failed, but it passed locally. --- If your project is set

[GitHub] spark issue #17297: [SPARK-14649][CORE] DagScheduler should not run duplicat...

2017-03-18 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17297 >> I don't think its true that it relaunches all tasks that hadn't completed when the fetch failure occurred. it relaunches all the tasks haven't completed, by the time the stage gets

[GitHub] spark issue #17334: [SPARK-19998][Block Manager]BlockRDD block not found Exc...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17334 **[Test build #3603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3603/testReport)** for PR 17334 at commit

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17342 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74790/ Test FAILed. ---

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17342 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #74790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74790/testReport)** for PR 17342 at commit

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106791067 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/StarJoinSuite.scala --- @@ -0,0 +1,488 @@ +/* + * Licensed to the

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106790932 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16330 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74788/ Test PASSed. ---

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16330 **[Test build #74788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74788/testReport)** for PR 16330 at commit

[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16626 **[Test build #74791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74791/testReport)** for PR 16626 at commit

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106790507 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SimpleCatalystConf.scala --- @@ -40,6 +40,9 @@ case class SimpleCatalystConf(

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106790475 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106790425 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #16626: [SPARK-19261][SQL] Alter add columns for Hive ser...

2017-03-18 Thread xwu0226
Github user xwu0226 commented on a diff in the pull request: https://github.com/apache/spark/pull/16626#discussion_r106790141 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -1860,4 +1861,119 @@ class HiveDDLSuite }

[GitHub] spark issue #17334: [SPARK-19998][Block Manager]BlockRDD block not found Exc...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17334 **[Test build #3603 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3603/testReport)** for PR 17334 at commit

[GitHub] spark issue #17311: [SPARK-19970][SQL] Table owner should be USER instead of...

2017-03-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17311 Hi, @vanzin and @srowen . Could you review this PR (again) when you have sometime? I feel guilty because this PR need to be verified by manually on kerberized clusters. --- If your

[GitHub] spark issue #17219: [SPARK-19876][SS][WIP] OneTime Trigger Executor

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17219 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17251 Hi, @cloud-fan . Is it possible that Spark 2.1.1 includes this fix? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17219: [SPARK-19876][SS][WIP] OneTime Trigger Executor

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74786/ Test PASSed. ---

[GitHub] spark issue #17219: [SPARK-19876][SS][WIP] OneTime Trigger Executor

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17219 **[Test build #74786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74786/testReport)** for PR 17219 at commit

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...

2017-03-18 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16330 Had a minor comment on the test case. LGTM otherwise and waiting for Jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #16330: [SPARK-18817][SPARKR][SQL] change derby log outpu...

2017-03-18 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/16330#discussion_r106789593 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -2909,6 +2910,30 @@ test_that("Collect on DataFrame when NAs exists at the top of a timestamp

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r106788877 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -212,8 +212,8 @@ case object TaskResultLost extends TaskFailedReason { *

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r106789380 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -540,6 +540,39 @@ class SparkContextSuite extends SparkFunSuite with

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r106788727 --- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala --- @@ -354,7 +354,7 @@ private[spark] object UIUtils extends Logging {

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r106789297 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -540,6 +540,39 @@ class SparkContextSuite extends SparkFunSuite with

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r106789004 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106789403 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106789422 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106789293 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE][WIP] spark-submit should han...

2017-03-18 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16596 Ok sounds good. Since this touches spark-submit scripts that are shared across all languages it would be good to get loop in some other reviewers as well. I can do that once we have the new diff

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106789103 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -83,9 +411,19 @@ object ReorderJoin extends

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106789008 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark issue #17342: [SPARK-18910][SPARK-12868] Allow adding jars from hdfs

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17342 **[Test build #74790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74790/testReport)** for PR 17342 at commit

[GitHub] spark pull request #17342: [SPARK-18910][SPARK-12868] Allow adding jars from...

2017-03-18 Thread weiqingy
GitHub user weiqingy opened a pull request: https://github.com/apache/spark/pull/17342 [SPARK-18910][SPARK-12868] Allow adding jars from hdfs ## What changes were proposed in this pull request? Spark 2.2 is going to be cut, it'll be great if SPARK-12868 can be resolved before

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106788869 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106788720 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on a diff in the pull request: https://github.com/apache/spark/pull/15363#discussion_r106788630 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala --- @@ -20,19 +20,347 @@ package

[GitHub] spark issue #17334: [SPARK-19998][Block Manager]BlockRDD block not found Exc...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17334 **[Test build #3602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3602/testReport)** for PR 17334 at commit

[GitHub] spark issue #17290: [SPARK-16599][CORE] java.util.NoSuchElementException: No...

2017-03-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17290 That's fine I can add a warning. I don't know if it is a bug situation but it sure could be. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request #17290: [SPARK-16599][CORE] java.util.NoSuchElementExcept...

2017-03-18 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/17290#discussion_r106788142 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala --- @@ -340,7 +340,7 @@ private[storage] class BlockInfoManager extends

[GitHub] spark issue #17286: [SPARK-19915][SQL] Exclude cartesian product candidates ...

2017-03-18 Thread nsyca
Github user nsyca commented on the issue: https://github.com/apache/spark/pull/17286 Right. I misread it. if there is no join predicate between a table and any cluster of tables, we should not consider that table in the join enumeration at all. We can simply push that table to be the

[GitHub] spark issue #16982: [SPARK-19654][SPARKR][SS] Structured Streaming API for R

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16982: [SPARK-19654][SPARKR][SS] Structured Streaming API for R

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74787/ Test PASSed. ---

[GitHub] spark issue #16982: [SPARK-19654][SPARKR][SS] Structured Streaming API for R

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #74787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74787/testReport)** for PR 16982 at commit

[GitHub] spark issue #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-18 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/17170 @felixcheung It is completely up to you. I'll have to patch one or another (and also #17218) and at the end of the day it doesn't make much difference. --- If your project is set up for it, you

[GitHub] spark issue #17286: [SPARK-19915][SQL] Exclude cartesian product candidates ...

2017-03-18 Thread ioana-delaney
Github user ioana-delaney commented on the issue: https://github.com/apache/spark/pull/17286 @gatorsmile Your example is correct. Given A J1 B J2 C: • level 0: (A), (B), (C) • level 1: {A, B}, ~{A, C}~, {B, C} • level 3: {A, B, C} Given A J1 B J2 C

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17246 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17246 **[Test build #74789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74789/testReport)** for PR 17246 at commit

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17246 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74789/ Test PASSed. ---

[GitHub] spark issue #17192: [SPARK-19849][SQL] Support ArrayType in to_json to produ...

2017-03-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17192 I will do another pass today and merge if others do not have more concerns? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17170 I suppose we should wait on #17321 before acting on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17286: [SPARK-19915][SQL] Exclude cartesian product candidates ...

2017-03-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17286 My example is not related to inequality join or equi join. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFi...

2017-03-18 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17274#discussion_r106787436 --- Diff: R/pkg/inst/tests/testthat/test_context.R --- @@ -177,6 +177,13 @@ test_that("add and get file to be downloaded with Spark job on every node",

[GitHub] spark issue #17246: [SPARK-19906][SS][DOCS] Documentation describing how to ...

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17246 **[Test build #74789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74789/testReport)** for PR 17246 at commit

[GitHub] spark issue #17286: [SPARK-19915][SQL] Exclude cartesian product candidates ...

2017-03-18 Thread nsyca
Github user nsyca commented on the issue: https://github.com/apache/spark/pull/17286 @gatorsmile An equality join in most cases has a better filtering than an inequality join. This can be used heuristically. However, this is not always true. An equality join can be a lookup join from

[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...

2017-03-18 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17274 test failed on appveyor - it looks like the connection to JVM is dead ``` . Error: add and get file to be downloaded with Spark job on every node (@test_context.R#184)

<    1   2   3   4   >