[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 When we turn on the option `true`, `t1` is broadcasted. ``` hive> set hive.auto.convert.join=true; hive> EXPLAIN SELECT /*+ MAPJOIN(t1) */ * FROM (select * from t1) x, t2; Warnin

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Since `set hive.auto.convert.join=true;` is default, I think mapjoin hint seems to work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 For the first case, I intentionally turn off the option to see the effect of the hint only. --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14132 For current hive, the `MAP_JOIN` hint is ignored totally right? How about old hive(or other databases that support MAP_JOIN or BROADCAST_JOIN hint)? I just wanna check the common rule to apply rel

[GitHub] spark pull request #14388: [SPARK-16362][SQL] Support ArrayType and StructTy...

2016-07-28 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/14388 [SPARK-16362][SQL] Support ArrayType and StructType in vectorized Parquet reader ## What changes were proposed in this pull request? Vectorization parquet reader now doesn't support complex

[GitHub] spark issue #14045: [SPARK-16362][SQL] Support ArrayType and StructType in v...

2016-07-28 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14045 Closed this PR in favor of the refactored one: #14388. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark issue #14182: [SPARK-16444][WIP][SparkR]: Isotonic Regression wrapper ...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62956/ Test PASSed. ---

[GitHub] spark pull request #14045: [SPARK-16362][SQL] Support ArrayType and StructTy...

2016-07-28 Thread viirya
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/14045 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #14182: [SPARK-16444][WIP][SparkR]: Isotonic Regression wrapper ...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14182 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14182: [SPARK-16444][WIP][SparkR]: Isotonic Regression wrapper ...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14182 **[Test build #62956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62956/consoleFull)** for PR 14182 at commit [`554e9a3`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14388: [SPARK-16362][SQL] Support ArrayType and StructType in v...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14388 **[Test build #62957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62957/consoleFull)** for PR 14388 at commit [`8cfeb7e`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 For `Table-Level Optimizer Hints` of MySQL, it applied to all table with the same name. > The hint applies to all tables that it names http://dev.mysql.com/doc/refman/5.7/en/optimizer-h

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

2016-07-28 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14365 cc @cloud-fan @yhuai @liancheng @rxin Please review this change. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, struct function should accept de...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14389 This should resolve the following two pull requests as well: https://github.com/apache/spark/pull/14353 https://github.com/apache/spark/pull/14374 --- If your project is set up for

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, struct function should ac...

2016-07-28 Thread petermaxlee
GitHub user petermaxlee opened a pull request: https://github.com/apache/spark/pull/14389 [SPARK-16714][SQL] map, struct function should accept decimals with different precision/scale ## What changes were proposed in this pull request? This patch changes the type coercion rule f

[GitHub] spark issue #14353: [SPARK-16714][SQL] `array` should create a decimal array...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14353 @dongjoon-hyun I created a patch here: https://github.com/apache/spark/pull/14389 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #14374: [SPARK-16735][SQL] `map` should create a decimal key or ...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14374 @biglobster @dongjoon-hyun I created a patch here: https://github.com/apache/spark/pull/14389 --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, struct function should accept de...

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14389 Could you add my testcase here, too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, struct function should accept de...

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14389 Also, please add `array` into the title and description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does no

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, struct function should accept de...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14389 **[Test build #62958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62958/consoleFull)** for PR 14389 at commit [`9774605`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14389 I was looking at the code, and I think this is a more general problem with decimal widening. The same problem exists for least, and other functions. ``` scala> sql("select least(0.1,

[GitHub] spark issue #36: Added a unit test for PairRDDFunctions.lookup

2016-07-28 Thread databricks-jenkins
Github user databricks-jenkins commented on the issue: https://github.com/apache/spark/pull/36 **[Test build #75 has started](https://jenkins.test.databricks.com/job/spark-pull-request-builder/75/consoleFull)** for PR 36 at commit [`66bab99`](https://github.com/apache/spark/commit/66

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on the issue: https://github.com/apache/spark/pull/14389 @dongjoon-hyun You only had one test case didn't you? I don't think that test case is useful since it was testing specifically checkInputDataTypes, which was not the right thing to test. Type co

[GitHub] spark pull request #14365: [SPARK-16628][SQL] Translate file-based relation ...

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14365#discussion_r72577696 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SchemaMapping.scala --- @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, array function should acc...

2016-07-28 Thread petermaxlee
Github user petermaxlee commented on a diff in the pull request: https://github.com/apache/spark/pull/14389#discussion_r72577751 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLTypeCoercionSuite.scala --- @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14389 Yea, for least and greatest, I opened this here https://github.com/apache/spark/pull/14294. Actually, I am worried if allowing lose of precision and fractions is okay. I first thought

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14389 @petermaxlee . Yep. I deleted my request, but you had better have a test case with real columns on table data. :) --- If your project is set up for it, you can reply to this email and have yo

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14389 cc @cloud-fan and @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark pull request #14390: [SPARK-15541] Casting ConcurrentHashMap to Concur...

2016-07-28 Thread maver1ck
GitHub user maver1ck opened a pull request: https://github.com/apache/spark/pull/14390 [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap ## What changes were proposed in this pull request? Casting ConcurrentHashMap to ConcurrentMap allows to run code compiled with Ja

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62959/consoleFull)** for PR 14390 at commit [`921e3c8`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62959/consoleFull)** for PR 14390 at commit [`921e3c8`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62959/ Test FAILed. ---

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #36: Added a unit test for PairRDDFunctions.lookup

2016-07-28 Thread databricks-jenkins
Github user databricks-jenkins commented on the issue: https://github.com/apache/spark/pull/36 **[Test build #75 has finished](https://jenkins.test.databricks.com/job/spark-pull-request-builder/75/consoleFull)** for PR 36 at commit [`66bab99`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14132 cc @rxin @hvanhovell what do you think of this case? `SELECT /*+ MAPJOIN(t1) */ * FROM (SELECT * FROM t1) x JOIN t1` Applying the hint to the `t1` after JOIN looks more intuitive to me.

[GitHub] spark pull request #14296: [SPARK-16639][SQL] The query with having conditio...

2016-07-28 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/14296#discussion_r72579088 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1207,6 +1207,17 @@ class Analyzer( v

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Thank you for giving opinion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62960/consoleFull)** for PR 14390 at commit [`3ffbff1`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14296 **[Test build #62961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62961/consoleFull)** for PR 14296 at commit [`f5e037a`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, array function should acc...

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/14389#discussion_r72579629 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -440,7 +440,7 @@ object TypeCoercion {

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 For old hives, I can get the binary after Hive 1.0. http://www-us.apache.org/dist/hive/ The above result was Hive 2.0 and 1.2. I think Hive 1.0 and 1.1 will be the same as expected

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62960/consoleFull)** for PR 14390 at commit [`3ffbff1`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62960/ Test FAILed. ---

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 For @cloud-fan 's choice, I think I can block the subqueries which are aliased by users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #14258: [Spark-16579][SparkR] add install_spark function

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14258 **[Test build #62962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62962/consoleFull)** for PR 14258 at commit [`f37a07c`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #14374: [SPARK-16735][SQL] `map` should create a decimal ...

2016-07-28 Thread biglobster
Github user biglobster commented on a diff in the pull request: https://github.com/apache/spark/pull/14374#discussion_r72580290 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala --- @@ -93,20 +93,45 @@ case class CreateMap(chi

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, array function should acc...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14389#discussion_r72580413 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -440,7 +440,7 @@ object TypeCoercion {

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, array function should acc...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14389#discussion_r72580646 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -440,7 +440,7 @@ object TypeCoercion {

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62963/consoleFull)** for PR 14390 at commit [`7bd4f44`](https://github.com/apache/spark/commit/7

[GitHub] spark pull request #14389: [SPARK-16714][SQL] map, array function should acc...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14389#discussion_r72580777 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -440,7 +440,7 @@ object TypeCoercion {

[GitHub] spark issue #14387: [SPARK-16764][SQL] Recommend disabling vectorized parque...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14387 **[Test build #62955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62955/consoleFull)** for PR 14387 at commit [`b62c1d2`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14387: [SPARK-16764][SQL] Recommend disabling vectorized parque...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14387 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62955/ Test PASSed. ---

[GitHub] spark issue #14387: [SPARK-16764][SQL] Recommend disabling vectorized parque...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14387 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14389 One problem of decimal type in Spark SQL is, the wider type of 2 decimal types may be illegal(exceed system limitation), then we have to truncate and suffer precision lose. This forces us to make

[GitHub] spark issue #36: Added a unit test for PairRDDFunctions.lookup

2016-07-28 Thread databricks-jenkins
Github user databricks-jenkins commented on the issue: https://github.com/apache/spark/pull/36 **[Test build #76 has started](https://jenkins.test.databricks.com/job/spark-pull-request-builder/76/consoleFull)** for PR 36 at commit [`0bdec01`](https://github.com/apache/spark/commit/0b

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Hmm, maybe more sophisticate method is needed for both the following cases. * SELECT /*+ MAPJOIN(t1) */ * FROM (SELECT * FROM t1) x JOIN t1 * SELECT /*+ MAPJOIN(t1) */ * FROM (SELEC

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14389 If we want to just fix these bugs, I think we should come up with a list about which functions(need arguments of same type) can accept precision lose and which can not. --- If your project is se

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14132 I agree with @cloud-fan. It is confusing that the hint defined in the top-level query gets applied to a subquery. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14389 FYI, I had a look before. Its map, array, greatest and least. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Thank you, @hvanhovell . Then should we block all subquery or allow the following cases? * SELECT /*+ MAPJOIN(t1) */ * FROM (SELECT * FROM t1) x JOIN t2; --- If your project is set up

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 At the implementation side, blocking all subqueries are easiest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14258: [Spark-16579][SparkR] add install_spark function

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62962/ Test PASSed. ---

[GitHub] spark issue #14355: [SPARK-16726][SQL] Improve `Union/Intersect/Except` erro...

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14355 Hi, @hvanhovell . Could you review this PR again when you have some time? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #14388: [SPARK-16362][SQL] Support ArrayType and StructType in v...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14388 **[Test build #62957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62957/consoleFull)** for PR 14388 at commit [`8cfeb7e`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14388: [SPARK-16362][SQL] Support ArrayType and StructType in v...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14388 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62957/ Test PASSed. ---

[GitHub] spark issue #14388: [SPARK-16362][SQL] Support ArrayType and StructType in v...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14388 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request #14391: [SPARK-16767][SQL] Add existsRecursively to UserD...

2016-07-28 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/14391 [SPARK-16767][SQL] Add existsRecursively to UserDefinedType ## What changes were proposed in this pull request? `DataType` has a method `existsRecursively` used to check recursively if a da

[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...

2016-07-28 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/14391 ping @cloud-fan Not sure if this is correct direction. Please let me know. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14391 **[Test build #62964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62964/consoleFull)** for PR 14391 at commit [`2ef1d19`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14132 `SELECT /*+ MAPJOIN(t1) */ * FROM (SELECT * FROM t1) x JOIN t2;` yea I think this one should be noop --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Thank you for guidance, @cloud-fan ! I'll implement like that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark pull request #14207: [SPARK-16552] [SQL] Store the Inferred Schemas in...

2016-07-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14207 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #14207: [SPARK-16552] [SQL] Store the Inferred Schemas into Exte...

2016-07-28 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14207 thanks, merging to master! cc @yhuai @liancheng we will address your comments in follow up PRs if you have any. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14389 **[Test build #62958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62958/consoleFull)** for PR 14389 at commit [`9774605`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14389 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62958/ Test PASSed. ---

[GitHub] spark issue #14389: [SPARK-16714][SQL] map, array function should accept dec...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14389 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14390 **[Test build #62963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62963/consoleFull)** for PR 14390 at commit [`7bd4f44`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62963/ Test FAILed. ---

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14296 **[Test build #62961 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62961/consoleFull)** for PR 14296 at commit [`f5e037a`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14296 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14296: [SPARK-16639][SQL] The query with having condition that ...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62961/ Test PASSed. ---

[GitHub] spark issue #36: Added a unit test for PairRDDFunctions.lookup

2016-07-28 Thread databricks-jenkins
Github user databricks-jenkins commented on the issue: https://github.com/apache/spark/pull/36 **[Test build #76 has finished](https://jenkins.test.databricks.com/job/spark-pull-request-builder/76/consoleFull)** for PR 36 at commit [`0bdec01`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14363 **[Test build #62965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62965/consoleFull)** for PR 14363 at commit [`97b0492`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #14390: [SPARK-15541] Casting ConcurrentHashMap to ConcurrentMap

2016-07-28 Thread maver1ck
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/14390 @jkbradley Could you look at it ? I think this is problem from: https://issues.apache.org/jira/browse/SPARK-10086 Maybe we should merge this PR to branch-1.6 before testing ? https:

[GitHub] spark issue #10909: [SPARK-10086] [MLlib] [Streaming] [PySpark] ignore Strea...

2016-07-28 Thread maver1ck
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/10909 @jkbradley What about merging this to branch-1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14132 **[Test build #62966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62966/consoleFull)** for PR 14132 at commit [`bd13473`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 Hi, the followings are updated. - Hints are not propagated into subqueries. - Add new testcases - `SELECT /*+ MAPJOIN(t1) */ * FROM t1 JOIN t1 JOIN t1 JOIN t1` (First `t1`)

[GitHub] spark issue #14376: [SPARK-16749][SQL] Simplify processing logic in LEAD/LAG...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14376 **[Test build #62967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62967/consoleFull)** for PR 14376 at commit [`cecbd8b`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14391 **[Test build #62964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62964/consoleFull)** for PR 14391 at commit [`2ef1d19`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14391 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14391 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62964/ Test PASSed. ---

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14363 **[Test build #62965 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62965/consoleFull)** for PR 14363 at commit [`97b0492`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14363 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14363 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62965/ Test FAILed. ---

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14132 **[Test build #62966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62966/consoleFull)** for PR 14132 at commit [`bd13473`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14132 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14132: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2016-07-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14132 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62966/ Test FAILed. ---

[GitHub] spark pull request #14392: [SPARK-16446] [SparkR] [ML] Gaussian Mixture Mode...

2016-07-28 Thread yanboliang
GitHub user yanboliang opened a pull request: https://github.com/apache/spark/pull/14392 [SPARK-16446] [SparkR] [ML] Gaussian Mixture Model wrapper in SparkR ## What changes were proposed in this pull request? Gaussian Mixture Model wrapper in SparkR, similarly to R's ```mvnorma

  1   2   3   4   >