[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220065589 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else {

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220065308 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1295,15 +1295,15 @@ object SQLConf { .booleanConf

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22198 **[Test build #96536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96536/testReport)** for PR 22198 at commit

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22198 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22198 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22198 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Conf to sp...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22540 Change itself LGTM except that bool comparison one --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread sandeep-katta
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 seems @cloud-fan comments are valid as it will not result in any behavior change, I will update the PR accordingly WDYT @srowen ---

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22541 LGTM, these functions have weird names and looks not very useful. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22507: [SPARK-25495][SS]FetchedData.reset should reset a...

2018-09-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22507#discussion_r220061053 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -874,6 +874,57 @@ abstract class

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22542 @cloud-fan Thanks a lot. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96531/ Test PASSed. ---

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22484 **[Test build #96531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96531/testReport)** for PR 22484 at commit

[GitHub] spark pull request #22507: [SPARK-25495][SS]FetchedData.reset should reset a...

2018-09-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22507#discussion_r220059919 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -874,6 +874,57 @@ abstract class

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220058070 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -131,11 +131,8 @@ object ArrowUtils { } else {

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 This is a behavior change and makes us different from Hive. However I can't find a strong reason to do it. It's like importing a database, but we can't automatically create table entries in the

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 Owp, I've been misreading that several times. Right. Well by analogy, if a database has a non default LOCATION then so do it's tables, and they are treated like EXTERNAL tables. Dropping the DB

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22541 I'm supportive of this change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220054697 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark pull request #22540: [SPARK-24324] [PYTHON] [FOLLOW-UP] Rename the Con...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22540#discussion_r220054060 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1295,15 +1295,15 @@ object SQLConf { .booleanConf

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22542 thanks, merging to master/2.4! @dilipbiswal sorry I didn't see your comment while merging. If the problem is about "implicit casting between two Map types", feel free to open a PR and

[GitHub] spark pull request #22542: [SPARK-25519][SQL] ArrayRemove function may retur...

2018-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22542 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22542 @cloud-fan @ueshin I would like to ask a question here. There is one more function i wanted to fix. Originally i wanted to do it as part of this PR.. then realized that its not as straight

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 > That link says Hive does support EXTERNAL. What am I missing? Hive supports `EXTERNAL` only for tables, not databases. The CREATE TABLE syntax: ``` CREATE [TEMPORARY]

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96529/ Test PASSed. ---

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22511 a possible approach: can we just not dispose the data in `TorrentBroadcast`? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22484 **[Test build #96529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96529/testReport)** for PR 22484 at commit

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96530/ Test PASSed. ---

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22484 **[Test build #96530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96530/testReport)** for PR 22484 at commit

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22541 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22541 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22539: [SPARK-25517][SQL] Detect/Infer date type in CSV file

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22539 Thank you for review, @HyukjinKwon . @softmanu . Could you take a look at [SPARK-19228](https://github.com/apache/spark/pull/21363) and close this PR and Apache Spark JIRA? ---

[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22511 The analysis makes sense to me. The thing I'm not sure is, how can we hit it? The "fetch block to temp file" code path is only enabled for big blocks (> 2GB). ---

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22541 **[Test build #96535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96535/testReport)** for PR 22541 at commit

[GitHub] spark issue #22541: [SPARK-23907][SQL] Revert regr_* functions entirely

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22541 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22521 nit. Could you add `[CORE]` to the PR title, too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/22521 yup; just did --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22521 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22521 **[Test build #96534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96534/testReport)** for PR 22521 at commit

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22521 @rxin . Could you fill the PR description, too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22521: [SPARK-24519] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_CO...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22521 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22539: [SPARK-25517][SQL] Detect/Infer date type in CSV file

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22539 I think this is a duplicate of https://github.com/apache/spark/pull/21363 --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22537: [SPARK-21291][R] add R partitionBy API in DataFra...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22537#discussion_r220050102 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -2713,8 +2713,16 @@ test_that("read/write text files", { expect_equal(colnames(df2),

[GitHub] spark pull request #22537: [SPARK-21291][R] add R partitionBy API in DataFra...

2018-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22537#discussion_r220050067 --- Diff: R/pkg/R/DataFrame.R --- @@ -2985,8 +2988,18 @@ setMethod("write.df", if (is.null(source)) { source <-

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 That link says Hive does support EXTERNAL. What am I missing? Well, in any event we aren't contemplating a behavior change here. If you delete a table with LOCATION specified, what should

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22484 BTW, I don't think we need `withTempTable` and `withTempPath` in this PR. Those are beyond of the scope of this PR. --- -

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/22542 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22484 Yep. This PR already is introducing new `trait` for `Benchmark` as a part of grand refactoring plan. I think the other new `trait`s also possible at this time. We will see. :) ---

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 > It will be great to explain how limit works in whole stage codegen, in general. This part is a little hard to understand and I believe many operators need to deal with limit as well. Ok.

[GitHub] spark issue #22494: [SPARK-25454][SQL] add a new config for picking minimum ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22494 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220048264 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark issue #22494: [SPARK-25454][SQL] add a new config for picking minimum ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22494 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22494: [SPARK-25454][SQL] add a new config for picking minimum ...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22494 **[Test build #96533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96533/testReport)** for PR 22494 at commit

[GitHub] spark pull request #22525: [SPARK-25503][CORE][WEBUI]Total task message in s...

2018-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22525 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22484 Thanks @dongjoon-hyun. except `withSQLConf`. We need `withTempTable` and `withTempPath`:

[GitHub] spark issue #22494: [SPARK-25454][SQL] add a new config for picking minimum ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22494 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22524 It will be great to explain how limit works in whole stage codegen, in general. This part is a little hard to understand. ---

[GitHub] spark issue #22525: [SPARK-25503][CORE][WEBUI]Total task message in stage pa...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22525 Merged to master/branch-2.4/branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22484 Also, cc @gatorsmile . Since we need `withSQLConf` in both Benchmark and TestSuite, I want to introduce `SupportWithSQLConf` trait . ---

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220046724 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220046213 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -73,14 +78,21 @@ public void append(InternalRow row) {

[GitHub] spark issue #22484: [SPARK-25476][SPARK-25510][TEST] Refactor AggregateBench...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22484 @wangyum . If you don't mind, could you review my PR to your branch, https://github.com/wangyum/spark/pull/11, which - deduplicates `withSQLConf` back - runs on AWS EC2 `r3.xlarge`?

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220046092 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -73,14 +78,21 @@ public void append(InternalRow row) {

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96527/ Test PASSed. ---

[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22429 **[Test build #96527 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96527/testReport)** for PR 22429 at commit

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044740 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22542 LGTM. The PR description has a typo: `ArrayPosition` => `ArrayRemove` --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044584 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -465,13 +465,18 @@ case class RangeExec(range:

[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22511 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96526/ Test PASSed. ---

[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22511 **[Test build #96526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96526/testReport)** for PR 22511 at commit

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044271 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044149 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -465,13 +465,18 @@ case class RangeExec(range:

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22542 cc @ueshin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22525: [SPARK-25503][CORE][WEBUI]Total task message in stage pa...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22525 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96523/ Test PASSed. ---

[GitHub] spark issue #22525: [SPARK-25503][CORE][WEBUI]Total task message in stage pa...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22525 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22525: [SPARK-25503][CORE][WEBUI]Total task message in stage pa...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22525 **[Test build #96523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96523/testReport)** for PR 22525 at commit

[GitHub] spark pull request #22484: [SPARK-25476][SPARK-25510][TEST] Refactor Aggrega...

2018-09-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22484#discussion_r220043531 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala --- @@ -0,0 +1,87 @@ +/* + * Licensed to

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220043421 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -73,14 +78,21 @@ public void append(InternalRow row) {

[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 yea, in Spark we conflate the two and treat a table as external if location is specified. However, Hive doesn't have external database, see:

[GitHub] spark issue #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.writeLega...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22453 **[Test build #96532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96532/testReport)** for PR 22453 at commit

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-24 Thread seancxmao
Github user seancxmao commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r220042478 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

[GitHub] spark issue #22535: [SPARK-17636][SQL][WIP] Parquet predicate pushdown in ne...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96528/ Test FAILed. ---

[GitHub] spark issue #22535: [SPARK-17636][SQL][WIP] Parquet predicate pushdown in ne...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22535 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22535: [SPARK-17636][SQL][WIP] Parquet predicate pushdown in ne...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22535 **[Test build #96528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96528/testReport)** for PR 22535 at commit

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220040370 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -38,6 +38,11 @@ protected int partitionIndex =

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22542 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96525/ Test PASSed. ---

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22542 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22542: [SPARK-25519][SQL] ArrayRemove function may return incor...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22542 **[Test build #96525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96525/testReport)** for PR 22542 at commit

[GitHub] spark issue #22316: [SPARK-25048][SQL] Pivoting by multiple columns in Scala...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22316 One safe change is to not use the `lit` function, but to do a manual pattern match and still use `Literal.apply`. We can investigate `Literal.create` in a followup ---

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220039084 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -38,6 +38,11 @@ protected int

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22325 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96522/ Test PASSed. ---

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

2018-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22325 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

2018-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22325 **[Test build #96522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96522/testReport)** for PR 22325 at commit

[GitHub] spark pull request #22453: [SPARK-20937][DOCS] Describe spark.sql.parquet.wr...

2018-09-24 Thread seancxmao
Github user seancxmao commented on a diff in the pull request: https://github.com/apache/spark/pull/22453#discussion_r220038438 --- Diff: docs/sql-programming-guide.md --- @@ -1002,6 +1002,21 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession

  1   2   3   4   5   >