[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208720757 Thanks. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208720658 Thanks. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14473][SQL] Define analysis rules to ca...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12246#issuecomment-208720591 **[Test build #55590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55590/consoleFull)** for PR 12246 at commit

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/12285#discussion_r59324092 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java --- @@ -69,7 +72,11 @@ public int numRecords() { return pos;

[GitHub] spark pull request: [SPARK-14548][SQL] Support not greater than an...

2016-04-11 Thread jliwork
Github user jliwork commented on a diff in the pull request: https://github.com/apache/spark/pull/12316#discussion_r59323956 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -1432,4 +1432,15 @@ class DataFrameSuite extends QueryTest with

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/12285#discussion_r59323954 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java --- @@ -255,6 +253,10 @@ public long spill(long size, MemoryConsumer

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/12285#discussion_r59323906 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java --- @@ -69,7 +72,11 @@ public int numRecords() { return pos;

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/12285#discussion_r59323828 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java --- @@ -69,7 +72,11 @@ public int numRecords() { return pos;

[GitHub] spark pull request: [SPARK-14513][CORE] Fix threads left behind af...

2016-04-11 Thread chtyim
Github user chtyim commented on the pull request: https://github.com/apache/spark/pull/12318#issuecomment-208718393 Addressed comment. Please have a look again. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/12285#issuecomment-208718277 That make sense, thanks for the explanation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12285#issuecomment-208718272 **[Test build #2777 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2777/consoleFull)** for PR 12285 at commit

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208717963 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208717961 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208717794 **[Test build #55586 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55586/consoleFull)** for PR 12322 at commit

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208717455 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208717457 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208717307 **[Test build #55585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55585/consoleFull)** for PR 12300 at commit

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread sitalkedia
Github user sitalkedia commented on the pull request: https://github.com/apache/spark/pull/12285#issuecomment-208717200 @davies Thanks for the explanation, your calculation makes sense. You are right that freeing the array can only make a difference of 15% in ideal case. But what we

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208716501 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-13568] [ML] Create feature transformer ...

2016-04-11 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/11601#discussion_r59323059 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala --- @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59323049 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -74,9 +75,18 @@ private[sql] class HiveSessionCatalog( // |

[GitHub] spark pull request: [SPARK-13568] [ML] Create feature transformer ...

2016-04-11 Thread hhbyyh
Github user hhbyyh commented on a diff in the pull request: https://github.com/apache/spark/pull/11601#discussion_r59323032 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala --- @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12321 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208715736 @srowen @zsxwing would you mind taking a look at this? Thanks! :-) --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12321#issuecomment-208715496 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208712945 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208712946 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208712642 **[Test build #55584 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55584/consoleFull)** for PR 12322 at commit

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208712262 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14549][ML][WIP] Copy the Vector and Mat...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12317#issuecomment-208712225 **[Test build #55589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55589/consoleFull)** for PR 12317 at commit

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208712263 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208712096 **[Test build #55583 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55583/consoleFull)** for PR 12323 at commit

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/12285#issuecomment-208710183 In your case, inside sorting, the key has 4 columns, the row has 6 columns, so each pair will need about 90 bytes, the array used by sort needs 16 bytes, so the memory

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread gatorsmile
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/12321#issuecomment-208710078 cc @andrewor14 @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-13792][SQL] Limit logging of bad record...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12173#issuecomment-208709983 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-13792][SQL] Limit logging of bad record...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12173#issuecomment-208709982 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-13792][SQL] Limit logging of bad record...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12173#issuecomment-208709842 **[Test build #55582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55582/consoleFull)** for PR 12173 at commit

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208706693 ok updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12321#issuecomment-208704967 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12321#issuecomment-208704966 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14362][SPARK-14406][SQL][Follow-up] DDL...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12321#issuecomment-208704828 **[Test build #55578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55578/consoleFull)** for PR 12321 at commit

[GitHub] spark pull request: [SPARK-14513][CORE] Fix threads left behind af...

2016-04-11 Thread chtyim
Github user chtyim commented on a diff in the pull request: https://github.com/apache/spark/pull/12318#discussion_r59320494 --- Diff: core/src/main/scala/org/apache/spark/HttpServer.scala --- @@ -155,6 +158,7 @@ private[spark] class HttpServer( throw new

[GitHub] spark pull request: [SPARK-14414] [SQL] improve the error message ...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12314#issuecomment-208703324 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14414] [SQL] improve the error message ...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12314#issuecomment-208703320 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-208703105 **[Test build #55588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55588/consoleFull)** for PR 12081 at commit

[GitHub] spark pull request: [SPARK-14414] [SQL] improve the error message ...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12314#issuecomment-208703085 **[Test build #55577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55577/consoleFull)** for PR 12314 at commit

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208703073 LGTM, could you update the description? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-14363] Fix executor OOM due to memory l...

2016-04-11 Thread sitalkedia
Github user sitalkedia commented on the pull request: https://github.com/apache/spark/pull/12285#issuecomment-208702898 @davies - Thanks for looking into it. I agree with you that its not a memory leak because that memory may be used later. However, not reducing the pointer array

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12079#issuecomment-208700359 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12079#issuecomment-208700356 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12079#issuecomment-208700160 **[Test build #55587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55587/consoleFull)** for PR 12079 at commit

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59319679 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -74,9 +75,18 @@ private[sql] class HiveSessionCatalog(

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59319585 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -121,8 +123,13 @@ class SessionCatalog(

[GitHub] spark pull request: [SPARK-14531][STREAMING] Flume streaming shoul...

2016-04-11 Thread yongtang
Github user yongtang commented on the pull request: https://github.com/apache/spark/pull/12305#issuecomment-208698736 Thanks @srowen. It looks like my previous understanding of the issue might be incorrect. Let me take a detailed look and see if I could update this pull request.

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread yongtang
Github user yongtang commented on the pull request: https://github.com/apache/spark/pull/12079#issuecomment-208697614 @holdenk The Scala implementation has ben completed in SPARK-13963. I updated the description of this pull request to show the linkage between this issue

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59319388 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -74,9 +75,18 @@ private[sql] class HiveSessionCatalog(

[GitHub] spark pull request: [SPARK-14554][SQL] disable whole stage codegen...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208696633 **[Test build #55586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55586/consoleFull)** for PR 12322 at commit

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12079#issuecomment-208696634 **[Test build #55587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55587/consoleFull)** for PR 12079 at commit

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread yongtang
Github user yongtang commented on a diff in the pull request: https://github.com/apache/spark/pull/12079#discussion_r59319063 --- Diff: python/pyspark/mllib/feature.py --- @@ -379,6 +379,17 @@ class HashingTF(object): """ def __init__(self, numFeatures=1 << 20):

[GitHub] spark pull request: [SPARK-14238][ML][MLLIB][PYSPARK] Add binary t...

2016-04-11 Thread yongtang
Github user yongtang commented on a diff in the pull request: https://github.com/apache/spark/pull/12079#discussion_r59318934 --- Diff: python/pyspark/ml/feature.py --- @@ -512,14 +512,19 @@ class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures, Java

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59318684 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala --- @@ -74,9 +75,18 @@ private[sql] class HiveSessionCatalog( // |

[GitHub] spark pull request: [SPARK-14544] [SQL] improve performance of SQL...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12311#issuecomment-208695373 **[Test build #2776 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2776/consoleFull)** for PR 12311 at commit

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208694952 **[Test build #55585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55585/consoleFull)** for PR 12300 at commit

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208694863 np, I already fixed it :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/12081#discussion_r59318540 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -121,8 +123,13 @@ class SessionCatalog(

[GitHub] spark pull request: [SPARK-14551][SQL] Reduce number of NameNode c...

2016-04-11 Thread rajeshbalamohan
Github user rajeshbalamohan commented on the pull request: https://github.com/apache/spark/pull/12319#issuecomment-208694717 Sure @rxin. makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-14544] [SQL] improve performance of SQL...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12311#issuecomment-208693950 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14544] [SQL] improve performance of SQL...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12311#issuecomment-208693949 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14544] [SQL] improve performance of SQL...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12311#issuecomment-208693827 **[Test build #55575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55575/consoleFull)** for PR 12311 at commit

[GitHub] spark pull request: [SPARK-14535][SQL] Remove buildInternalScan fr...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12300#issuecomment-208693757 Ah. sorry. Maybe the commit I just made caused the conflicts... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59318208 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -161,24 +163,27 @@ private[state]

[GitHub] spark pull request: [SPARK-14549][ML][WIP] Copy the Vector and Mat...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12317#issuecomment-208693649 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14549][ML][WIP] Copy the Vector and Mat...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12317#issuecomment-208693648 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14549][ML][WIP] Copy the Vector and Mat...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12317#issuecomment-208693559 **[Test build #55574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55574/consoleFull)** for PR 12317 at commit

[GitHub] spark pull request: SPARK-14551. [SQL] Reduce number of NN calls i...

2016-04-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12319#issuecomment-208693509 Two other things: 1. can you follow the format used by other prs, i.e. [SPARK-14551][SQL] ... 2. can you spell out name node? Most people don't know that NN =

[GitHub] spark pull request: [SPARK-14132][SPARK-14133][SQL] Alter table pa...

2016-04-11 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12220 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-14513][CORE] Fix threads left behind af...

2016-04-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12318#discussion_r59318036 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -350,4 +352,10 @@ private[spark] object JettyUtils extends Logging {

[GitHub] spark pull request: [SPARK-14513][CORE] Fix threads left behind af...

2016-04-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12318#discussion_r59318015 --- Diff: core/src/main/scala/org/apache/spark/HttpServer.scala --- @@ -155,6 +158,7 @@ private[spark] class HttpServer( throw new

[GitHub] spark pull request: [SPARK-14132][SPARK-14133][SQL] Alter table pa...

2016-04-11 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/12220#issuecomment-208693034 Merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317875 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -506,7 +512,6 @@ private[state]

[GitHub] spark pull request: [SPARK-14554][SQL] Dataset.map may generate wr...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208693002 **[Test build #55584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55584/consoleFull)** for PR 12322 at commit

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12323#issuecomment-208692992 **[Test build #55583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55583/consoleFull)** for PR 12323 at commit

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317846 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala --- @@ -22,12 +22,12 @@ import scala.reflect.ClassTag

[GitHub] spark pull request: [SPARK-14459] [SQL] Detect relation partitioni...

2016-04-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12239#issuecomment-208692890 cc @liancheng and @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
Github user lw-lin commented on a diff in the pull request: https://github.com/apache/spark/pull/12323#discussion_r59317781 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala --- @@ -26,12 +26,11 @@ private[streaming] class

[GitHub] spark pull request: [SPARK-14556][SQL] Code clean-ups for package ...

2016-04-11 Thread lw-lin
GitHub user lw-lin opened a pull request: https://github.com/apache/spark/pull/12323 [SPARK-14556][SQL] Code clean-ups for package o.a.s.sql.execution.streaming.state ## What changes were proposed in this pull request? - `StateStoreConf.**max**DeltasForSnapshot` was

[GitHub] spark pull request: [SPARK-14554][SQL] Dataset.map may generate wr...

2016-04-11 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12322#issuecomment-208692370 cc @davies @marmbrus @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-14554][SQL] Dataset.map may generate wr...

2016-04-11 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/12322 [SPARK-14554][SQL] Dataset.map may generate wrong java code for wide table ## What changes were proposed in this pull request? In

[GitHub] spark pull request: [SPARK-14513][CORE] Fix threads left behind af...

2016-04-11 Thread chtyim
Github user chtyim commented on a diff in the pull request: https://github.com/apache/spark/pull/12318#discussion_r59317619 --- Diff: core/src/main/scala/org/apache/spark/HttpServer.scala --- @@ -155,6 +158,7 @@ private[spark] class HttpServer( throw new

[GitHub] spark pull request: [SPARK-14545][SQL] Improve `LikeSimplification...

2016-04-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12312#discussion_r59317551 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/LikeSimplificationSuite.scala --- @@ -61,6 +61,20 @@ class

[GitHub] spark pull request: [SPARK-14545][SQL] Improve `LikeSimplification...

2016-04-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/12312#discussion_r59317529 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -529,6 +530,9 @@ object LikeSimplification extends

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-208691858 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-208691859 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-208691760 **[Test build #55573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55573/consoleFull)** for PR 12081 at commit

[GitHub] spark pull request: [SPARK-14547] Avoid DNS resolution for reusing...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12315#issuecomment-208691652 **[Test build #2775 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2775/consoleFull)** for PR 12315 at commit

[GitHub] spark pull request: SPARK-14551. [SQL] Reduce number of NN calls i...

2016-04-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12319#issuecomment-208691389 @rajeshbalamohan can you fix your title / description? You are having the title spilling over to the end of the description. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-14547] Avoid DNS resolution for reusing...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12315#issuecomment-208690975 **[Test build #2774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2774/consoleFull)** for PR 12315 at commit

[GitHub] spark pull request: [SPARK-14555] First cut of Python API for Stru...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12320#issuecomment-208690430 **[Test build #55581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55581/consoleFull)** for PR 12320 at commit

[GitHub] spark pull request: [SPARK-14555] First cut of Python API for Stru...

2016-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12320#issuecomment-208690449 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-13792][SQL] Limit logging of bad record...

2016-04-11 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12173#issuecomment-208690324 **[Test build #55582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55582/consoleFull)** for PR 12173 at commit

  1   2   3   4   5   6   7   >