[GitHub] spark pull request #14638: [SPARK-11374][SQL] Support `skip.header.line.coun...

2016-11-29 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14638#discussion_r90098551 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -122,10 +126,20 @@ class HadoopTableReader( val attrsWithIndex =

[GitHub] spark issue #15910: [SPARK-18476][SPARKR][ML]:SparkR Logistic Regression sho...

2016-11-29 Thread wangmiao1981
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/15910 @yanboliang @felixcheung I am back from vacation and made changes according to your comments. Thanks! --- If your project is set up for it, you can reply to this email and have your r

[GitHub] spark pull request #14638: [SPARK-11374][SQL] Support `skip.header.line.coun...

2016-11-29 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14638#discussion_r90098793 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -122,10 +126,20 @@ class HadoopTableReader( val attrsWithIndex =

[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...

2016-11-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15975#discussion_r90098792 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -76,9 +76,6 @@ class JDBCOptions(

[GitHub] spark issue #16064: [SPARK-18633][ML][Example]: Add multiclass logistic regr...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16064 **[Test build #69344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69344/consoleFull)** for PR 16064 at commit [`7040089`](https://github.com/apache/spark/commit/7

[GitHub] spark pull request #16044: [Spark-18614][SQL] Incorrect predicate pushdown f...

2016-11-29 Thread nsyca
Github user nsyca commented on a diff in the pull request: https://github.com/apache/spark/pull/16044#discussion_r90100457 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -932,7 +932,7 @@ object PushPredicateThroughJoin extends

[GitHub] spark pull request #14638: [SPARK-11374][SQL] Support `skip.header.line.coun...

2016-11-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/14638#discussion_r90101517 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -122,10 +126,20 @@ class HadoopTableReader( val attrsWith

[GitHub] spark issue #15979: [SPARK-18251][SQL] the type of Dataset can't be Option o...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15979 FWIW I don't think we should call it nonflat. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16048 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16048 **[Test build #69345 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69345/consoleFull)** for PR 16048 at commit [`9ff2ed4`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15982 **[Test build #69337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69337/consoleFull)** for PR 15982 at commit [`2e03ee6`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69337/ Test PASSed. ---

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15982 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16064: [SPARK-18633][ML][Example]: Add multiclass logistic regr...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16064 **[Test build #69344 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69344/consoleFull)** for PR 16064 at commit [`7040089`](https://github.com/apache/spark/commit/

[GitHub] spark issue #16064: [SPARK-18633][ML][Example]: Add multiclass logistic regr...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16064 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16064: [SPARK-18633][ML][Example]: Add multiclass logistic regr...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16064 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69344/ Test PASSed. ---

[GitHub] spark pull request #15924: [SPARK-18498] [SQL] Revise HDFSMetadataLog API fo...

2016-11-29 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/15924#discussion_r90090753 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala --- @@ -129,48 +129,18 @@ class HDFSMetadataLog[T <: AnyRe

[GitHub] spark pull request #15954: [WIP][SPARK-18516][SQL] Split state and progress ...

2016-11-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/15954#discussion_r90108129 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala --- @@ -669,55 +658,48 @@ trait StreamTest extends QueryTest with SharedSQLCo

[GitHub] spark issue #15954: [WIP][SPARK-18516][SQL] Split state and progress in stre...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69346/consoleFull)** for PR 15954 at commit [`d9d8f82`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #16038: [SPARK-18471][CORE] New treeAggregate overload for big l...

2016-11-29 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16038 Without understanding the specifics of the ML part here - wont the actual impact of a large dense vector on Task 'bytes' be minimal at best ? We do compress the task binary; and 1B zero's should

[GitHub] spark pull request #16065: [SPARK-17064][SQL] Changed ExchangeCoordinator re...

2016-11-29 Thread markhamstra
GitHub user markhamstra opened a pull request: https://github.com/apache/spark/pull/16065 [SPARK-17064][SQL] Changed ExchangeCoordinator re-partitioning to avoid additional data … ## What changes were proposed in this pull request? Re-partitioning logic in ExchangeCoordin

[GitHub] spark pull request #15924: [SPARK-18498] [SQL] Revise HDFSMetadataLog API fo...

2016-11-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15924 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16065: [SPARK-17064][SQL] Changed ExchangeCoordinator re-partit...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16065 **[Test build #69347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69347/consoleFull)** for PR 16065 at commit [`561fcf6`](https://github.com/apache/spark/commit/5

[GitHub] spark pull request #16063: [SPARK-18622][SQL] Remove TypeCoercion rules for ...

2016-11-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/16063#discussion_r90110048 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -482,21 +482,6 @@ object TypeCoercion {

[GitHub] spark issue #16065: [SPARK-17064][SQL] Changed ExchangeCoordinator re-partit...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16065 Wrong JIRA ticket? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or i

[GitHub] spark issue #16065: [SPARK-18631][SQL] Changed ExchangeCoordinator re-partit...

2016-11-29 Thread markhamstra
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/16065 @rxin fixed it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #15505: [SPARK-17931][CORE] taskScheduler has some unneeded seri...

2016-11-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/15505 I agree with Kay that putting in a smaller change first is better, assuming it still has the performance gains. That doesn't preclude any further optimizations that are bigger changes. I'm

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15982 **[Test build #69348 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69348/consoleFull)** for PR 15982 at commit [`8ac9276`](https://github.com/apache/spark/commit/8

[GitHub] spark pull request #15954: [WIP][SPARK-18516][SQL] Split state and progress ...

2016-11-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/15954#discussion_r90112136 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala --- @@ -38,11 +40,11 @@ trait StreamingQuery { def name: String

[GitHub] spark pull request #14638: [SPARK-11374][SQL] Support `skip.header.line.coun...

2016-11-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/14638#discussion_r90112125 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -113,6 +113,10 @@ class HadoopTableReader( val tablePath =

[GitHub] spark pull request #16066: [SPARK-18632][SQL] AggregateFunction should not i...

2016-11-29 Thread hvanhovell
GitHub user hvanhovell opened a pull request: https://github.com/apache/spark/pull/16066 [SPARK-18632][SQL] AggregateFunction should not implement ImplicitCastInputTypes ## What changes were proposed in this pull request? `AggregateFunction` currently implements `ImplicitCastInp

[GitHub] spark issue #16066: [SPARK-18632][SQL] AggregateFunction should not implemen...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16066 **[Test build #69349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69349/consoleFull)** for PR 16066 at commit [`9a722cf`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16066: [SPARK-18632][SQL] AggregateFunction should not implemen...

2016-11-29 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16066 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if th

[GitHub] spark issue #16065: [SPARK-18631][SQL] Changed ExchangeCoordinator re-partit...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16065 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

[GitHub] spark pull request #16066: [SPARK-18632][SQL] AggregateFunction should not i...

2016-11-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16066#discussion_r90114430 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala --- @@ -56,6 +52,20 @@ case class Last(child: Expression,

[GitHub] spark issue #16066: [SPARK-18632][SQL] AggregateFunction should not implemen...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16066 LGTM other than that tiny comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-11-29 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r90114390 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala --- @@ -408,14 +411,18 @@ class HiveCommandSuite extends QueryTes

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-11-29 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r90092813 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -922,6 +923,29 @@ private[spark] class HiveExternalCatalog(conf:

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-11-29 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r90103384 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -189,11 +189,28 @@ abstract class ExternalCatalog {

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-11-29 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r90092477 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -922,6 +923,29 @@ private[spark] class HiveExternalCatalog(conf:

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

2016-11-29 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15998#discussion_r90097773 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -482,6 +482,19 @@ class InMemoryCatalog( }

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

2016-11-29 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15998 * looks good once InMemoryCatalog is fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15877 Thanks - I'm going to merge this in master. I will submit a follow-up PR to simplify this a little bit, and remove the handling of float/double/decimal types and require explicit user action on how to

[GitHub] spark pull request #15877: [SPARK-18429] [SQL] implement a new Aggregate for...

2016-11-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15877 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #16063: [SPARK-18622][SQL] Remove TypeCoercion rules for Average...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16063 **[Test build #69343 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69343/consoleFull)** for PR 16063 at commit [`7596b5a`](https://github.com/apache/spark/commit/

[GitHub] spark issue #16063: [SPARK-18622][SQL] Remove TypeCoercion rules for Average...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16063 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69343/ Test FAILed. ---

[GitHub] spark issue #16063: [SPARK-18622][SQL] Remove TypeCoercion rules for Average...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16063 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16062: [SPARK-18629][SQL] Fix numPartition of JDBCSuite Testcas...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16062 **[Test build #69340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69340/consoleFull)** for PR 16062 at commit [`30c5d6f`](https://github.com/apache/spark/commit/

[GitHub] spark issue #16062: [SPARK-18629][SQL] Fix numPartition of JDBCSuite Testcas...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16062 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16062: [SPARK-18629][SQL] Fix numPartition of JDBCSuite Testcas...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16062 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69340/ Test PASSed. ---

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Inference Rul...

2016-11-29 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/16067 [SPARK-17897] [SQL] Fixed IsNotNull Inference Rule ### What changes were proposed in this pull request? The `constraints` of an operator is the expressions that evaluate to `true` for all th

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16048 **[Test build #69351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69351/consoleFull)** for PR 16048 at commit [`27102eb`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16067 **[Test build #69350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69350/consoleFull)** for PR 16067 at commit [`0722ae5`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #16044: [Spark-18614][SQL] Incorrect predicate pushdown from Exi...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16044 **[Test build #69341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69341/consoleFull)** for PR 16044 at commit [`d4002c7`](https://github.com/apache/spark/commit/

[GitHub] spark pull request #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scal...

2016-11-29 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16009#discussion_r90120095 --- Diff: docs/ml-features.md --- @@ -1188,7 +1188,9 @@ categorical features. The number of bins is set by the `numBuckets` parameter. I that the number

[GitHub] spark issue #16044: [Spark-18614][SQL] Incorrect predicate pushdown from Exi...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69341/ Test PASSed. ---

[GitHub] spark issue #16044: [Spark-18614][SQL] Incorrect predicate pushdown from Exi...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16044 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15877: [SPARK-18429] [SQL] implement a new Aggregate for CountM...

2016-11-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15877 Hey guys - after looking at the pr more, I'm afraid we have gone overboard with testing here. Most of the test cases written are just repeating each other and doing exactly the same thing. For testing

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69352/consoleFull)** for PR 15954 at commit [`aa8af9c`](https://github.com/apache/spark/commit/a

[GitHub] spark pull request #15877: [SPARK-18429] [SQL] implement a new Aggregate for...

2016-11-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15877#discussion_r90122127 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAggSuite.scala --- @@ -0,0 +1,320 @@ +/* + * Li

[GitHub] spark pull request #14638: [SPARK-11374][SQL] Support `skip.header.line.coun...

2016-11-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/14638#discussion_r90123854 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -113,6 +113,10 @@ class HadoopTableReader( val tabl

[GitHub] spark issue #15946: [SPARK-18513][Structured Streaming] Record and recover w...

2016-11-29 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15946 @lw-lin could you close this one please? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69353/consoleFull)** for PR 15954 at commit [`c11d2e5`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #15780: [SPARK-18284][SQL] Make ExpressionEncoder.serializer.nul...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15780 **[Test build #69342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69342/consoleFull)** for PR 15780 at commit [`39e4930`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15780: [SPARK-18284][SQL] Make ExpressionEncoder.serializer.nul...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15780 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69342/ Test PASSed. ---

[GitHub] spark issue #15780: [SPARK-18284][SQL] Make ExpressionEncoder.serializer.nul...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15780 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69352/consoleFull)** for PR 15954 at commit [`aa8af9c`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69352/ Test FAILed. ---

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request #15982: [SPARK-18546][core] Fix merging shuffle spills wh...

2016-11-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15982#discussion_r90127514 --- Diff: core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java --- @@ -75,13 +75,6 @@ @Mock(answer = RETURNS_SMART_NULLS)

[GitHub] spark pull request #15982: [SPARK-18546][core] Fix merging shuffle spills wh...

2016-11-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15982#discussion_r90126726 --- Diff: core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java --- @@ -40,9 +41,11 @@ import org.mockito.stubbing.Answer;

[GitHub] spark pull request #15982: [SPARK-18546][core] Fix merging shuffle spills wh...

2016-11-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15982#discussion_r90121766 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java --- @@ -337,42 +340,47 @@ void forceSorterToSpill() throws IOException {

[GitHub] spark pull request #15982: [SPARK-18546][core] Fix merging shuffle spills wh...

2016-11-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15982#discussion_r90127615 --- Diff: core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java --- @@ -86,14 +88,7 @@ public int compare(

[GitHub] spark issue #16065: [SPARK-18631][SQL] Changed ExchangeCoordinator re-partit...

2016-11-29 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/16065 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69353/consoleFull)** for PR 15954 at commit [`c11d2e5`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69353/ Test FAILed. ---

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16067 **[Test build #69354 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69354/consoleFull)** for PR 16067 at commit [`f693040`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90129155 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -1089,66 +1064,10 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90116879 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -1016,11 +1013,6 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)]) /

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r88075283 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapRedCommitProtocol.scala --- @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Soft

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90122251 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90121527 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90120144 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90127987 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90129259 --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala --- @@ -561,7 +561,7 @@ class PairRDDFunctionsSuite extends SparkFunSuite with

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r88077635 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90121670 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r87708046 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90119536 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request #15861: [SPARK-18294][CORE] Implement commit protocol to ...

2016-11-29 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/15861#discussion_r90124359 --- Diff: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala --- @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark issue #15861: [SPARK-18294][CORE] Implement commit protocol to support...

2016-11-29 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15861 @jiangxb1987 I did a single pass review - particularly given the similarities in both the codepaths and the classnames, I will need to go over it again to ensure we dont miss anything. --- If your

[GitHub] spark issue #15954: [SPARK-18516][SQL] Split state and progress in streaming

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15954 **[Test build #69355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69355/consoleFull)** for PR 15954 at commit [`69d9b4a`](https://github.com/apache/spark/commit/6

[GitHub] spark pull request #15954: [SPARK-18516][SQL] Split state and progress in st...

2016-11-29 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/15954#discussion_r90130320 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala --- @@ -64,23 +68,26 @@ trait StreamingQuery { /**

[GitHub] spark issue #16066: [SPARK-18632][SQL] AggregateFunction should not implemen...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16066 **[Test build #69356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69356/consoleFull)** for PR 16066 at commit [`1246792`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15255 **[Test build #69357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69357/consoleFull)** for PR 15255 at commit [`57817a1`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16048 **[Test build #69345 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69345/consoleFull)** for PR 16048 at commit [`9ff2ed4`](https://github.com/apache/spark/commit/

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69345/ Test FAILed. ---

[GitHub] spark pull request #15954: [SPARK-18516][SQL] Split state and progress in st...

2016-11-29 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/15954#discussion_r90132677 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -59,13 +62,20 @@ class StreamingQueryManager private[sql]

<    1   2   3   4   5   6   7   >