[GitHub] spark pull request: [SPARK-2817] [SQL] add show create table sup...

2014-08-12 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1760#issuecomment-51876493 Be aware that the `udf_unix_timestamp` case is timezone sensitive. That's why we reset timezone to America/Los_Angeles in `beforeAll`. This may be related to your test

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/1889#discussion_r16096763 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala --- @@ -372,7 +372,7 @@ object MapType { * The

[GitHub] spark pull request: [MLlib] Correctly set vectorSize and alpha

2014-08-12 Thread Ishiihara
GitHub user Ishiihara opened a pull request: https://github.com/apache/spark/pull/1900 [MLlib] Correctly set vectorSize and alpha You can merge this pull request into a Git repository by running: $ git pull https://github.com/Ishiihara/spark Word2Vec-bugfix Alternatively you

[GitHub] spark pull request: [MLlib] Correctly set vectorSize and alpha

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1900#issuecomment-51876764 QA tests have started for PR 1900. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18361/consoleFull ---

[GitHub] spark pull request: [SPARK-2650][SQL] More precise initial buffer ...

2014-08-12 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/1901 [SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can

[GitHub] spark pull request: [SPARK-2650][SQL] Build column buffers in smal...

2014-08-12 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1880#issuecomment-51877140 Opened #1901 for precise initial buffer size estimation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1889#discussion_r16097043 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala --- @@ -372,7 +372,7 @@ object MapType { * The

[GitHub] spark pull request: [SPARK-2650][SQL] More precise initial buffer ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1901#issuecomment-51877331 QA tests have started for PR 1901. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18362/consoleFull ---

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/1889#discussion_r16097181 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala --- @@ -372,7 +372,7 @@ object MapType { * The

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51877969 QA tests have started for PR 1889. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18363/consoleFull ---

[GitHub] spark pull request: [SPARK-2817] [SQL] add show create table sup...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1760#issuecomment-51878493 QA results for PR 1760:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2981][GraphX] EdgePartition1D Int overf...

2014-08-12 Thread larryxiao
GitHub user larryxiao opened a pull request: https://github.com/apache/spark/pull/1902 [SPARK-2981][GraphX] EdgePartition1D Int overflow minor fix You can merge this pull request into a Git repository by running: $ git pull https://github.com/larryxiao/spark 2981

[GitHub] spark pull request: [MLlib] Correctly set vectorSize and alpha

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1900#issuecomment-51879862 QA results for PR 1900:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2981][GraphX] EdgePartition1D Int overf...

2014-08-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1902#issuecomment-51879950 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51879912 QA results for PR 1891:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2643] Stages web ui has ERROR when pool...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1854#issuecomment-51881015 QA tests have started for PR 1854. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18364/consoleFull ---

[GitHub] spark pull request: [SPARK-2650][SQL] More precise initial buffer ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1901#issuecomment-51881563 QA results for PR 1901:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [MLlib] Correctly set vectorSize and alpha

2014-08-12 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1900#issuecomment-51882028 LGTM. Merged into both master and branch-1.1. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [MLlib] Correctly set vectorSize and alpha

2014-08-12 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1900 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51882621 QA results for PR 1889:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1897#discussion_r16099170 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala --- @@ -185,6 +185,58 @@ class LogisticRegressionSuite

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-12 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1897#discussion_r16099253 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala --- @@ -137,11 +154,45 @@ abstract class

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51883319 The build failure was caused by PySpark. (Let me try again...) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51883335 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51883703 QA tests have started for PR 1891. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18365/consoleFull ---

[GitHub] spark pull request: [SPARK-2924] remove default args to overloaded...

2014-08-12 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1704#issuecomment-51884393 Hm - seems like it's either out of data again or it didn't get merged. If you bring it up to date I _promise_ I'll merge it quickly (or I'll do a manual resolution if

[GitHub] spark pull request: [SPARK-2643] Stages web ui has ERROR when pool...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1854#issuecomment-51884838 QA results for PR 1854:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51885042 Thanks a lot for triggering test... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51885640 QA tests have started for PR 1889. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18366/consoleFull ---

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-08-12 Thread liyezhang556520
Github user liyezhang556520 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r16100690 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +193,93 @@ private class MemoryStore(blockManager:

[GitHub] spark pull request: [SPARK-2947] DAGScheduler resubmit the stage i...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1877#issuecomment-51887280 QA tests have started for PR 1877. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18367/consoleFull ---

[GitHub] spark pull request: [SPARK-2062][GraphX] VertexRDD.apply does not ...

2014-08-12 Thread larryxiao
GitHub user larryxiao opened a pull request: https://github.com/apache/spark/pull/1903 [SPARK-2062][GraphX] VertexRDD.apply does not use the mergeFunc create verticesDeduplicate with reduceByKey, using mergeFunc then proceed with verticesDedup But this is not tested and

[GitHub] spark pull request: [SPARK-2062][GraphX] VertexRDD.apply does not ...

2014-08-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1903#issuecomment-51887950 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2878]: Fix custom spark.kryo.registrato...

2014-08-12 Thread GrahamDennis
Github user GrahamDennis commented on the pull request: https://github.com/apache/spark/pull/1890#issuecomment-51888165 I've updated my PR, and now instead of getting the Executor process to download jars before registering itself with the application driver, the Worker process

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51889573 QA results for PR 1891:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2986] [SQL] fixed: setting properties d...

2014-08-12 Thread guowei2
GitHub user guowei2 opened a pull request: https://github.com/apache/spark/pull/1904 [SPARK-2986] [SQL] fixed: setting properties does not effect it seems that set command does not run by SparkSQLDriver. it runs on hive api. user can not change reduce number by setting

[GitHub] spark pull request: [SPARK-2924] remove default args to overloaded...

2014-08-12 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/1704#discussion_r16103132 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala --- @@ -98,9 +98,11 @@ class StreamingContext private[streaming] (

[GitHub] spark pull request: [SPARK-2986] [SQL] fixed: setting properties d...

2014-08-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1904#issuecomment-51891698 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2924] remove default args to overloaded...

2014-08-12 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/1704#discussion_r16103171 --- Diff: project/MimaExcludes.scala --- @@ -133,7 +136,10 @@ object MimaExcludes {

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51891864 QA results for PR 1889:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2947] DAGScheduler resubmit the stage i...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1877#issuecomment-51892023 QA results for PR 1877:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2970] [SQL] spark-sql script ends with ...

2014-08-12 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/1891#issuecomment-51896285 Ah... It's strange. I think there are no test using SparkSQLCLIDriver right? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...

2014-08-12 Thread hunggpham
Github user hunggpham commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-51898209 Yes i forked your repository and can see the codes now.One question: I don't see backprop codes.Will that be added later? Thanks. Sent from cell

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51898757 QA tests have started for PR 1889. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18368/consoleFull ---

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/1905 [SPARK-2987] - Adjust build system and tests to work with scala 2.11 This patch assumes that we are on akka 2.3.4. So it does not deal with that. You can merge this pull request into a Git

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51900317 QA tests have started for PR 1905. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18369/consoleFull ---

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51900326 QA results for PR 1905:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51900706 QA tests have started for PR 1905. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18370/consoleFull ---

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51900759 QA results for PR 1905:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51901484 QA tests have started for PR 1905. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18371/consoleFull ---

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51903449 QA tests have started for PR 1905. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18372/consoleFull ---

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51904098 QA results for PR 1889:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51906898 QA results for PR 1905:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-51909284 QA results for PR 1905:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51918543 @ueshin If we are changing the default value of `ArrayType.containsNull` to `true`, we may also need to add logic to set that value based on data sources. In

[GitHub] spark pull request: [WIP][SPARK-1720]use LD_LIBRARY_PATH instead o...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51920856 QA tests have started for PR 1031. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18373/consoleFull ---

[GitHub] spark pull request: [WIP][SPARK-1720]use LD_LIBRARY_PATH instead o...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51921344 QA results for PR 1031:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [WIP][SPARK-1720] Add LD_LIBRARY_PATH environm...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51923013 QA tests have started for PR 1031. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18374/consoleFull ---

[GitHub] spark pull request: [WIP][SPARK-1720] Add the value of LD_LIBRARY_...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51931255 QA results for PR 1031:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-1766] sorted functions to meet pedantic...

2014-08-12 Thread copester
GitHub user copester opened a pull request: https://github.com/apache/spark/pull/1906 [SPARK-1766] sorted functions to meet pedantic requirements You can merge this pull request into a Git repository by running: $ git pull https://github.com/ResilientScience/spark master

[GitHub] spark pull request: [SPARK-1766] sorted functions to meet pedantic...

2014-08-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1906#issuecomment-51932732 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2140] Enabling YARN memory overhead cal...

2014-08-12 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/1906#discussion_r16121764 --- Diff: yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -113,11 +113,12 @@ class Client(clientArgs: ClientArguments, hadoopConf:

[GitHub] spark pull request: SPARK-1297 Upgrade HBase dependency to 0.98

2014-08-12 Thread tmalaska
Github user tmalaska commented on the pull request: https://github.com/apache/spark/pull/1893#issuecomment-51936244 +1 I'm all for the update to 0.98 HBase. Just make sure we address everything that Sean O is asking. We need this to be able to build with Hadoop1 or

[GitHub] spark pull request: [SPARK-2987][WIP] - Adjust build system and te...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-5193 QA tests have started for PR 1905. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18375/consoleFull ---

[GitHub] spark pull request: [SPARK-2924] remove default args to overloaded...

2014-08-12 Thread avati
Github user avati commented on the pull request: https://github.com/apache/spark/pull/1704#issuecomment-51942842 Had conflict with recent merge of 9038d9. Addressed other code comments as well. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [WIP][SPARK-1720] Add the value of LD_LIBRARY_...

2014-08-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51944309 Does this work cross-platform? I think OSX uses `DYLD_LIBRARY_PATH`, and Windows probably uses something else. --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-2969][SQL] Make ScalaReflection be able...

2014-08-12 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1889#issuecomment-51944634 It would be great to get this change in for 1.1 since its kinda of an API change. However, for now I'm okay if trying to store nulls into parquet arrays throws

[GitHub] spark pull request: [WIP][SPARK-1720] Add the value of LD_LIBRARY_...

2014-08-12 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51945400 It's definitely not cross-platform. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [WIP][SPARK-1720] Add the value of LD_LIBRARY_...

2014-08-12 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-51946148 Yes,I only took into account the linux and windows. Yarn only distinguish windows and non-windows at [ApplicationConstants.java#L210](

[GitHub] spark pull request: Vida/ec2 reuse security group

2014-08-12 Thread vidaha
Github user vidaha commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-51949860 Hi Josh, IMHO, it's best not to require a Spark cluster name and the security group to be the same. While you can reuse an existing security group to launch

[GitHub] spark pull request: Vida/ec2 reuse security group

2014-08-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-51953660 After a closer look, it looks like the existing `spark-ec2` already names instances after `cluster_name`, so it's safe and backwards-compatible to use that to identify

[GitHub] spark pull request: SPARK-2333

2014-08-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-51954787 Jenkins, this is ok to test. Test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Add unit test to spark_ec2 script

2014-08-12 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/134#issuecomment-51955535 This PR is a pretty neat idea; it would be great to get more testing of the Spark EC2 scripts, especially as we add new features (like #1899). Does anyone want to

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1907 [WIP] [SPARK-2468] Netty based block server / client module This is a rough draft, but it can't be much worse than the old Netty module since the old one didn't really work :) Compared with

[GitHub] spark pull request: SPARK-2333

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-51955708 QA tests have started for PR 1899. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18376/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51956457 QA tests have started for PR 1907. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18377/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51956530 QA results for PR 1907:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957070 @rxin Very nice ! Do you have any benchmarks of how fast things are ? In terms of say % of network bandwidth we can use ? --- If your project is set up for it, you can

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957157 QA tests have started for PR 1907. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18378/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16132782 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957233 That's part of the TODO :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957274 QA results for PR 1907:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2333

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-51957963 QA tests have started for PR 1899. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18379/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51958686 QA tests have started for PR 1907. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18380/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51958814 QA results for PR 1907:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread dcapwell
Github user dcapwell commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16133840 --- Diff: core/src/main/scala/org/apache/spark/network/netty/ReferenceCountedBuffer.scala --- @@ -17,15 +17,25 @@ package

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134006 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread dcapwell
Github user dcapwell commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134103 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client.scala --- @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-08-12 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/1360#issuecomment-51959659 @pwendell Still should go into 1.1.0... The change is fairly small, and the unpatched behavior is pretty nasty for long-running applications. --- If your project

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134163 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: SPARK-2830

2014-08-12 Thread atalwalkar
GitHub user atalwalkar opened a pull request: https://github.com/apache/spark/pull/1908 SPARK-2830 As per discussions with Xiangrui, I've reorganized and edited the mllib documentation. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: SPARK-2425 Don't kill a still-running Applicat...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1360#issuecomment-51960128 QA tests have started for PR 1360. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18382/consoleFull ---

[GitHub] spark pull request: SPARK-2830

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1908#issuecomment-51960126 QA tests have started for PR 1908. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18381/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134481 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134600 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134921 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -262,77 +271,49 @@ object BlockFetcherIterator {

[GitHub] spark pull request: SPARK-2830

2014-08-12 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1908#issuecomment-51961773 @atalwalkar Could you add `[MLLIB] re-organize mllib documentation` to the title? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: SPARK-2830

2014-08-12 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/1908#issuecomment-51962565 nit: That's not really an adequate title for this PR, Ameet. It should include enough description so that we can tell what it is about in the corresponding subject

[GitHub] spark pull request: [WIP][SPARK-2935][SQL]Fix parquet predicate pu...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1863#issuecomment-51962859 QA tests have started for PR 1863. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18383/consoleFull ---

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16135674 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client.scala --- @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

  1   2   3   >