[GitHub] spark issue #18779: [SPARK-21580][SQL]There's a bug with `Group by ordinal`

2017-07-30 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 Thanks,i will update @viirya --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18778 Do you want to create a new test suite for that under https://github.com/apache/spark/tree/master/core/src/test/java/test/org/apache/spark? The following suite seems to be irrelevant

[GitHub] spark pull request #18659: [SPARK-21404][PYSPARK][WIP] Simple Python Vectori...

2017-07-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18659#discussion_r130276299 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java --- @@ -65,15 +65,44 @@ final Row row;

[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18106 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80066/ Test FAILed. ---

[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18106 **[Test build #80066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80066/testReport)** for PR 18106 at commit

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 Sure, let me try soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18780 @HyukjinKwon , can you please also add the closed PR title `closes #xxx `, that would be easier for us to identify, otherwise we need to click on by one, thanks! --- If your project is

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18778 `JavaSparkContext` is the JAVA friendly `SparkContext`. Is that possible the JAVA programs could still directly use the `SparkContext` APIs. I am afraid these changes could break them. --- If

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 I guess It should be good to identify important and good PRs but I am sure most of them were inactive to review comments or Jenkins failures more than a month and I left a comment to check if

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18778 **[Test build #80069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80069/testReport)** for PR 18778 at commit

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18778 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18779: [SPARK-21580][SQL]There's a bug with `Group by ordinal`

2017-07-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 BTW, we should modify the PR description too. Please briefly describe the problem, what the fix is. Thanks. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18778 Thank you for review, @gatorsmile . Do you mean

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 I will go over the SQL-related PRs in the next few days to identify which PRs should not be closed. We should find somebody else to take them over. --- If your project is set up for it, you can

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 But wouldn't we still be able to look though the PRs and make the list even after being closed here? I think I am lost about why we should leave them open. --- If your project is set up for

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18724 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80065/ Test FAILed. ---

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18724 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18724 **[Test build #80065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80065/testReport)** for PR 18724 at commit

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 First, we need to identify a list of PRs that need to be taken over. For example, in SQL PRs, https://github.com/apache/spark/pull/18515 should be fixed, even if the original PR creator might

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 @gatorsmile, Sure, I will take them out if any ML committer suggests. cc @jkbradley, @yanboliang and @mengxr who I know. Will also take a look and check them as far as I can once more.

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 BTW, I suggest to let the committers who knew the component very well to give the suggestion to close the PRs. For example, I will not suggest to close any ML PR. --- If your project is set up

[GitHub] spark pull request #18779: [SPARK-21580][SQL]There's a bug with `Group by or...

2017-07-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18779#discussion_r130272987 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1010,7 +1014,16 @@ class Analyzer(

[GitHub] spark pull request #18779: [SPARK-21580][SQL]There's a bug with `Group by or...

2017-07-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18779#discussion_r130272963 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1010,7 +1014,16 @@ class Analyzer(

[GitHub] spark pull request #18779: [SPARK-21580][SQL]There's a bug with `Group by or...

2017-07-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18779#discussion_r130272850 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1010,7 +1014,16 @@ class Analyzer(

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 The number of JIRAs is too many. To be honest, I do not have a bandwidth to go over all the in-progress JIRAs. I suggest to review the inactive PRs one more time to ensure we do not miss any

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18768 ok, I'll make a pr. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 ok. When I finished this pr, I'll do. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18768 Yes! Thanks for your efforts! Please update the documentation at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18772 Let us do it in a separate PR. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 ok, I'll fix that way. Is it okay to include the fix in this pr? or, follow-up? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18772 https://github.com/apache/spark/pull/18772#issuecomment-318851145 This sounds a good idea to me. 👍 --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18768 @viirya I've no idea about a way to add tests for this kind of cases and I think the previous prs similar to this case (just add code for warning) also had no test. But, if we could, we'd better to

[GitHub] spark issue #18779: [SPARK-21580][SQL]There's a bug with `Group by ordinal`

2017-07-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 A specified title for this might be better. Such as "Integers in aggregation expressions are wrongly taken as group-by ordinal". --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 Is the point to leave them open to make reviewers read? I think we already had few discussion about this. The (I think) last resort, automatic closing, was already suggested before - @rxin.

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 If the PRs are closed, we might not read them any more. Thus, my suggestion is be careful when you want to close the PRs. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #16365: [SPARK-18950][SQL] Report conflicting fields when mergin...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16365 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16365: [SPARK-18950][SQL] Report conflicting fields when mergin...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16365 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80064/ Test PASSed. ---

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 @gatorsmile, I think we still could find others even after being closed if the point is to prevent forgetting. This list remains and there are many in-progress JIRAs we could find someone to

[GitHub] spark issue #16365: [SPARK-18950][SQL] Report conflicting fields when mergin...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16365 **[Test build #80064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80064/testReport)** for PR 16365 at commit

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18780 Before closing these PRs, we need to figure out whether they should be taken over by someone else. If we simply close them, we might forget to fix the related issues. --- If your project is

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 oh, yea. Sorry to forget to add `@`... ;) Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18780 **[Test build #80068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80068/testReport)** for PR 18780 at commit

[GitHub] spark issue #18731: [SPARK-20990][SQL] Read all JSON documents in files when...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18731 BTW, you also need to add a few error handling test cases to ensure the corrupted record handling works. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #18731: [SPARK-20990][SQL] Read all JSON documents in fil...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18731#discussion_r130270513 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -347,13 +347,18 @@ class JacksonParser(

[GitHub] spark issue #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18780 cc @srowen, @jiangxb1987, @gatorsmile, @ueshin, @jerryshao, @vanzin, @cloud-fan and @MLnick who I believe are interested in this and could double check this list. --- If your project is set

[GitHub] spark pull request #18780: [INTRA] Close stale PRs

2017-07-30 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/18780 [INTRA] Close stale PRs ## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with #18017 Closes #11494 Closes

[GitHub] spark issue #18475: [SPARK][Tests] assert messager not right

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18475 Sounds we should close if it is not the very first contribution of the author. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #18779: [SPARK-21580][SQL]There's a bug with `Group by ordinal`

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80067/testReport)** for PR 18779 at commit

[GitHub] spark pull request #18779: [SPARK-21580][SQL]There's a bug with `Group by or...

2017-07-30 Thread 10110346
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/18779 [SPARK-21580][SQL]There's a bug with `Group by ordinal` ## What changes were proposed in this pull request? create temporary view data as select * from values (1, 1), (1, 2),

[GitHub] spark issue #18609: Spark SQL merge small files to big files Update InsertIn...

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18609 @wuzhilon, could you explain why it is problematic if we just repartition? I didn't understand > the problem is unable to know the number of data MB, I can only get the number of

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18772 ping me? : ) Will review it in the next few days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

2017-07-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #18768: [SPARK-21533][SQL] Print warning messages when ov...

2017-07-30 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18768#discussion_r130268285 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -66,6 +66,8 @@ private[sql] class HiveSessionCatalog( *

[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18106 **[Test build #80066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80066/testReport)** for PR 18106 at commit

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18668 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18668 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80063/ Test PASSed. ---

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18668 **[Test build #80063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80063/testReport)** for PR 18668 at commit

[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18106 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18778 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18778 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80062/ Test FAILed. ---

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18778 **[Test build #80062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80062/testReport)** for PR 18778 at commit

[GitHub] spark pull request #14151: [SPARK-16496][SQL] Add wholetext as option for re...

2017-07-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/14151#discussion_r130267182 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFileWholeTextReader.scala --- @@ -0,0 +1,57 @@ +/* + * Licensed

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

2017-07-30 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/18664 I agree with @icexelloss that we should use `SQLConf.SESSION_LOCAL_TIMEZONE` so I sent a pr to use it to @BryanCutler. And hopefully we should use it for `df.collect()` and `df.toPandas` as well.

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18724 **[Test build #80065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80065/testReport)** for PR 18724 at commit

[GitHub] spark pull request #18724: [SPARK-21519][SQL] Add an option to the JDBC data...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18724#discussion_r130266625 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -135,6 +135,8 @@ class JDBCOptions(

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18724 Could you add a test case to `JDBCSuite`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18724: [SPARK-21519][SQL] Add an option to the JDBC data source...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18724 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #14151: [SPARK-16496][SQL] Add wholetext as option for re...

2017-07-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/14151#discussion_r130265982 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFileWholeTextReader.scala --- @@ -0,0 +1,57 @@ +/* + * Licensed

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18778 Could you check whether JAVA APIs still work? Could you add the related test cases? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread yaooqinn
Github user yaooqinn commented on the issue: https://github.com/apache/spark/pull/18668 @gatorsmile CliSuite will get nothing configured here because the cliSessionState is not reuse as we expected, see at

[GitHub] spark pull request #18775: [SPARK-21575][SparkR] Eliminate needless synchron...

2017-07-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18775 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18775: [SPARK-21575][SparkR] Eliminate needless synchronization...

2017-07-30 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/18775 Merged into master, thanks for all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18421: [SPARK-21213][SQL] Support collecting partition-level st...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18421 ping @mbasmanova --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18765: [SPARK-19720][CORE] Redact sensitive information from Sp...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18765 This sounds reasonable to backport to 2.1. First, please update your PR title with [BACKPORT-2.1] Second, please clean your PR description and also explain it is a backport PR at

[GitHub] spark issue #18697: [SPARK-16683][SQL] Repeated joins to same table can leak...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18697 I will review this next week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18765: [SPARK-19720][CORE] Redact sensitive information from Sp...

2017-07-30 Thread dmvieira
Github user dmvieira commented on the issue: https://github.com/apache/spark/pull/18765 I'm sorry... I was just suggesting it because is a major issue as described here: https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-19720 I'm using airflow for job submit

[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14151 cc @jiangxb1987 @gengliangwang Could you review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request #16365: [SPARK-18950][SQL] Report conflicting fields when...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16365#discussion_r130261853 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala --- @@ -184,14 +184,17 @@ class DataTypeSuite extends SparkFunSuite

[GitHub] spark pull request #16365: [SPARK-18950][SQL] Report conflicting fields when...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16365#discussion_r130261821 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala --- @@ -469,9 +469,16 @@ object StructType extends AbstractDataType {

[GitHub] spark pull request #16365: [SPARK-18950][SQL] Report conflicting fields when...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/16365#discussion_r130261747 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala --- @@ -469,9 +469,16 @@ object StructType extends AbstractDataType {

[GitHub] spark issue #16365: [SPARK-18950][SQL] Report conflicting fields when mergin...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16365 **[Test build #80064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80064/testReport)** for PR 16365 at commit

[GitHub] spark issue #16365: [SPARK-18950][SQL] Report conflicting fields when mergin...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16365 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18765: [SPARK-19720][CORE] Redact sensitive information from Sp...

2017-07-30 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18765 Should we backport this to 2.1 since it's a major bugfix(as described in the JIRA)? @vanzin @srowen --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #17180: [SPARK-19839][Core]release longArray in BytesToBy...

2017-07-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17180 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18668 **[Test build #80063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80063/testReport)** for PR 18668 at commit

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17180 Thanks! Merging to master. @zhzhan Could you address the comments about the test case in the follow-up PR? Thanks! --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18668 If this is an issue in `SparkSQLCliDriver `, could you add a test case to `CliSuite`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #18684: [SPARK-21475][Core] Use NIO's Files API to replace FileI...

2017-07-30 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18684 @cloud-fan @JoshRosen can you please help to review, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

2017-07-30 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/17180 This seems fine to me. That said, the updated test case is a bit confusing, but I don't think the original test was too clear to begin with. The original test was using the `iterator()`

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-30 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18711 In the past days we often assume that in standalone mode, if we don't set the "spark.executor.cores" property, then we should have no more than one executor per host, but this PR explains a

[GitHub] spark issue #13067: [SPARK-4131] [SQL] Support INSERT OVERWRITE [LOCAL] DIRE...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/13067 This sounds a syntactic sugar. Users can achieve it by creating an external table and then inserting the data. Another way is to use our DataFrameWriter API. Please let me know whether

[GitHub] spark issue #18692: [SPARK-21417][SQL] Detect joind conditions via filter ex...

2017-07-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18692 BTW, your PR title and descriptions are out of dated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request #18776: [MINOR] Minor comment fixes in merge_spark_pr.py ...

2017-07-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18776 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18776: [MINOR] Minor comment fixes in merge_spark_pr.py script

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18776 Merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18776: [MINOR] Minor comment fixes in merge_spark_pr.py script

2017-07-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18776 I guess it is safe to merge. Thanks for approval @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-30 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18711 @srowen due to the current design of standalone cluster manager, if we don't set `--total-executor-cores`, then Spark application will try to acquire all the free cores on this cluster, and it

[GitHub] spark issue #18778: [SPARK-21578][CORE] Consolidate redundant SparkContext c...

2017-07-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18778 **[Test build #80062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80062/testReport)** for PR 18778 at commit

[GitHub] spark pull request #18767: [SPARK-21572][Spark Shell] Add description on how...

2017-07-30 Thread davidxdh
Github user davidxdh closed the pull request at: https://github.com/apache/spark/pull/18767 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18767: [SPARK-21572][Spark Shell] Add description on how to exi...

2017-07-30 Thread davidxdh
Github user davidxdh commented on the issue: https://github.com/apache/spark/pull/18767 OK. I close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

  1   2   >