[GitHub] spark pull request: [WIP][SPARK-12069][SQL] Update documentation w...

2015-12-08 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10060#discussion_r47000161 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala --- @@ -73,7 +73,25 @@ class TypedColumn[-T, U]( /** * :: Experimental :

[GitHub] spark pull request: [SPARK-12165][SPARK-12189] Fix bugs in evictio...

2015-12-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10170#discussion_r47000196 --- Diff: core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala --- @@ -110,6 +110,12 @@ private[spark] class UnifiedMemoryManager priva

[GitHub] spark pull request: SPARK-11193 - Hack to support SynchronizedMap ...

2015-12-08 Thread jbonofre
Github user jbonofre commented on the pull request: https://github.com/apache/spark/pull/10203#issuecomment-162987492 There are two things: - as you said, we could use a ConcurrentHashMap in KinesisReceiver - but it still means that users can't use Kryo serializer with Synchron

[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6502#issuecomment-162987332 **[Test build #47355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47355/consoleFull)** for PR 6502 at commit [`ce52797`](https://gith

[GitHub] spark pull request: [SPARK-12202][REPL] Parse Scala command line a...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10199#issuecomment-162986919 **[Test build #47352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47352/consoleFull)** for PR 10199 at commit [`faef98d`](https://gi

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162986697 **[Test build #47353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47353/consoleFull)** for PR 10204 at commit [`207895b`](https://gi

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46999417 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala --- @@ -281,6 +296,202 @@ class HistoryServerSuite extends SparkFunSuite

[GitHub] spark pull request: SPARK-11193 - Hack to support SynchronizedMap ...

2015-12-08 Thread EugenCepoi
Github user EugenCepoi commented on the pull request: https://github.com/apache/spark/pull/10203#issuecomment-162985706 Why don't you just replace the use of SynchronizedMap in KinesisReceiver with a ConcurrentHashMap instead? --- If your project is set up for it, you can reply to th

[GitHub] spark pull request: [SPARK-3873] [build] Add style checker to enfo...

2015-12-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6502#issuecomment-162985841 not that it should matter, but retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...

2015-12-08 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-162984975 For Unsafe.getInt, what's the difference between ON_HEAP or OFF_HEAP? --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-12202][REPL] Parse Scala command line a...

2015-12-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/10199#issuecomment-162984950 Isn't this the same as #9824? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46998222 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala --- @@ -146,22 +136,51 @@ class HistoryServer( override def stop() {

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162984299 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-12165][SPARK-12189] Fix bugs in evictio...

2015-12-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10170#discussion_r46997970 --- Diff: core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala --- @@ -100,7 +100,7 @@ private[spark] class UnifiedMemoryManager privat

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46997833 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -678,6 +827,54 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46997864 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -678,6 +827,54 @@ private[history] class FsHistoryProvider(conf:

[GitHub] spark pull request: [SPARK-7727] [SQL] Avoid inner classes in Rule...

2015-12-08 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10174#discussion_r46997422 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -17,6 +17,8 @@ package org.apache.spark.sql.c

[GitHub] spark pull request: [SPARK-7727] [SQL] Avoid inner classes in Rule...

2015-12-08 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10174#discussion_r46997515 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala --- @@ -56,12 +55,16 @@ abstract class RuleExecutor[TreeType <

[GitHub] spark pull request: [SPARK-12165][SPARK-12189] Fix bugs in evictio...

2015-12-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10170#discussion_r46997361 --- Diff: core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala --- @@ -128,9 +132,11 @@ class StorageMemoryPool(lock: Object) extends Memo

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162983041 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...

2015-12-08 Thread ted-yu
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-162982916 @rxin @zsxwing FYI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162983043 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12202][REPL] Parse Scala command line a...

2015-12-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10199#issuecomment-162982937 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request: [SPARK-11563] [core] [repl] Use RpcEnv to tran...

2015-12-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9923#issuecomment-162982557 > IIUC, http server is useless once we remove akka? There's the http-based broadcast server, which IIRC is not the default. Does anyone even use that these days?

[GitHub] spark pull request: [SPARK-11031][SPARKR] Method str() on a DataFr...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9613#issuecomment-162982218 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-11031][SPARKR] Method str() on a DataFr...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9613#issuecomment-162982220 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [RFC] Add type-safe config keys to SparkConf.

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10205#issuecomment-162981891 **[Test build #47350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47350/consoleFull)** for PR 10205 at commit [`8b00d76`](https://gi

[GitHub] spark pull request: [SPARK-11031][SPARKR] Method str() on a DataFr...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9613#issuecomment-162981999 **[Test build #47349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47349/consoleFull)** for PR 9613 at commit [`8140e20`](https://git

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10202 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-11031][SPARKR] Method str() on a DataFr...

2015-12-08 Thread olarayej
Github user olarayej commented on a diff in the pull request: https://github.com/apache/spark/pull/9613#discussion_r46995766 --- Diff: R/pkg/R/DataFrame.R --- @@ -2151,3 +2151,97 @@ setMethod("coltypes", rTypes }) + +#' Display the struc

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10202#issuecomment-162979118 LGTM. Thank you for the fix! I am merging it to master and branch 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46995495 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,579 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-11563] [core] [repl] Use RpcEnv to tran...

2015-12-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9923#discussion_r46995428 --- Diff: core/src/main/scala/org/apache/spark/rpc/netty/NettyStreamManager.scala --- @@ -41,7 +42,10 @@ private[netty] class NettyStreamManager(rpcEnv: Netty

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46995405 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala --- @@ -33,7 +33,35 @@ private[spark] case class ApplicationAtt

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46995458 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala --- @@ -73,4 +101,17 @@ private[history] abstract class Applic

[GitHub] spark pull request: [SPARK-11605] [MLlib] ML 1.6 QA: API: Java com...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10102#issuecomment-162977622 **[Test build #2185 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2185/consoleFull)** for PR 10102 at commit [`5ef6ff8`](https://g

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162977371 **[Test build #2184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2184/consoleFull)** for PR 10166 at commit [`cb33653`](https://

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46995257 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162978022 ```dev/lint-python``` should catch these issues --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If you

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46995031 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [RFC] Add type-safe config keys to SparkConf.

2015-12-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/10205#issuecomment-162977465 I'm leaving this as an RFC to start with; if people like it, I'll probably just attach this to SPARK-529 (after reopening it). /cc @andrewor14 @rxin @srowen -

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162976307 That was a spurious test failure; I asked it to retest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [RFC] Add type-safe config keys to SparkConf.

2015-12-08 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/10205 [RFC] Add type-safe config keys to SparkConf. This is, in a way, the basics to enable SPARK-529 (which was closed as won't fix but I think is still valuable). In fact, Spark SQL created somet

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162976755 **[Test build #2184 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2184/consoleFull)** for PR 10166 at commit [`cb33653`](https://g

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162976343 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread aray
Github user aray commented on the pull request: https://github.com/apache/spark/pull/10202#issuecomment-162975930 @yhuai can you take a look at this small patch to pivot? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162975883 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10204#issuecomment-162975881 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10202#issuecomment-162975379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10202#issuecomment-162975377 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12153][MLlib]add support of arbitrary l...

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/10152#issuecomment-162975357 Overall, I'd say it's unclear whether we need to modify our implementation. How about we look for use cases and see if people have reported differences between follo

[GitHub] spark pull request: [SPARK-12165][SPARK-12189] Fix bugs in evictio...

2015-12-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10170#discussion_r46994235 --- Diff: core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala --- @@ -73,27 +73,31 @@ class StorageMemoryPool(lock: Object) extends Memor

[GitHub] spark pull request: [SPARK-9819][Streaming][Documentation] Clarify...

2015-12-08 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8103#issuecomment-162974987 LGTM. ping @tdas to take a final look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request: [SPARK-11031][SPARKR] Method str() on a DataFr...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9613#issuecomment-162975173 **[Test build #47349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47349/consoleFull)** for PR 9613 at commit [`8140e20`](https://gith

[GitHub] spark pull request: [SPARK-12165][SPARK-12189] Fix bugs in evictio...

2015-12-08 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/10170#discussion_r46994074 --- Diff: core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala --- @@ -73,27 +73,31 @@ class StorageMemoryPool(lock: Object) extends Memor

[GitHub] spark pull request: [SPARK-12205][SQL] Pivot fails Analysis when a...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10202#issuecomment-162975139 **[Test build #47343 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47343/consoleFull)** for PR 10202 at commit [`8a3f555`](https://g

[GitHub] spark pull request: [SPARK-12153][MLlib]add support of arbitrary l...

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/10152#issuecomment-162974386 @MLnick Oops, you're right; we are completely ignoring sentence boundaries. I didn't look carefully enough at the code. I'll correct my comment above too. >

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46993661 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Softwa

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-12-08 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8512#issuecomment-162974033 Has anyone looked at the performance of this versus S3a in Hadoop 2.7+? Because while I do agree this will dramatically improve s3n: and s3: perf, all ongoing Hado

[GitHub] spark pull request: [SPARK-12153][MLlib]add support of arbitrary l...

2015-12-08 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/10152#discussion_r46992991 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -469,7 +469,32 @@ class Word2VecModel private[spark] ( this(Wo

[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8551 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-12160][MLLIB] Use SQLContext.getOrCreat...

2015-12-08 Thread jkbradley
Github user jkbradley closed the pull request at: https://github.com/apache/spark/pull/10183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-162970671 LGTM. Merging with master and branch-1.6 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-12-08 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8512#discussion_r46992376 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala --- @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162970048 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-12-08 Thread piaozhexiu
Github user piaozhexiu commented on a diff in the pull request: https://github.com/apache/spark/pull/8512#discussion_r46992201 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala --- @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] spark pull request: [SPARK-12078][Core]Fix ByteBuffer.limit misuse

2015-12-08 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/10076 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] spark pull request: [SPARK-12159] [ML] Add user guide section for ...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10166#issuecomment-162970052 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12188] [SQL] Code refactoring and comme...

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10184 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12063][SQL] Use number in group by clau...

2015-12-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10052#issuecomment-162970002 We'll have to note the change in the release notes, but since its a no-op to sort by a constant I think we can safely change behavior here. --- If your project is set

[GitHub] spark pull request: [SPARK-12078][Core]Fix ByteBuffer.limit misuse

2015-12-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10076#discussion_r46992341 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -253,7 +253,7 @@ private[spark] class Executor( val directRes

[GitHub] spark pull request: [SPARK-12188] [SQL] Code refactoring and comme...

2015-12-08 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10184#discussion_r46991908 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -67,15 +67,21 @@ class Dataset[T] private[sql]( tEncoder: Encoder[T]) ex

[GitHub] spark pull request: [SPARK-12188] [SQL] Code refactoring and comme...

2015-12-08 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/10184#discussion_r46992054 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -429,18 +432,18 @@ class Dataset[T] private[sql]( /** * (Jav

[GitHub] spark pull request: [SPARK-12188] [SQL] Code refactoring and comme...

2015-12-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10184#issuecomment-162969383 Thanks, I'm going to merge this to master and 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If y

[GitHub] spark pull request: [SQL] Try to run execution hive's derby in mem...

2015-12-08 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/10204 [SQL] Try to run execution hive's derby in memory. This PR tries to make execution hive's derby run in memory since it is a fake metastore and every time we create a HiveContext, we will switch to a

[GitHub] spark pull request: [SPARK-11605] [MLlib] ML 1.6 QA: API: Java com...

2015-12-08 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/10102#issuecomment-162968904 Thank you for updating! LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your p

[GitHub] spark pull request: [SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk l...

2015-12-08 Thread steveloughran
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/8512#discussion_r46991325 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkS3Util.scala --- @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-12161][SQL] Ignore order of predicates ...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10163#issuecomment-162967445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12161][SQL] Ignore order of predicates ...

2015-12-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10163#issuecomment-162967443 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12161][SQL] Ignore order of predicates ...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10163#issuecomment-162967261 **[Test build #47342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47342/consoleFull)** for PR 10163 at commit [`c81aa46`](https://g

[GitHub] spark pull request: [SPARK-12199][DOC] Follow-up: Refine example c...

2015-12-08 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/10193#issuecomment-162967326 @yinxusen Could you fix conflicts? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12195] [SQL] Adding BigDecimal, Date an...

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10188 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12195] [SQL] Adding BigDecimal, Date an...

2015-12-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10188#issuecomment-162966881 Thanks, merging to master and 1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-12201][SQL] add type coercion rule for ...

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10196 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12201][SQL] add type coercion rule for ...

2015-12-08 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/10196#issuecomment-162965801 Thanks, merging to master and 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9468][Yarn][Core] Avoid scheduling task...

2015-12-08 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/7786#issuecomment-162965678 Jerry: do you know when YARN visits this decisions about having things pre-emptible? That is: if you are given a warning dos that mean the container will go any ti

[GitHub] spark pull request: [SPARK-12136] [STREAMING] rddToFileName does n...

2015-12-08 Thread bomeng
Github user bomeng commented on a diff in the pull request: https://github.com/apache/spark/pull/10185#discussion_r46990153 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala --- @@ -892,12 +892,14 @@ object StreamingContext extends Logging {

[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...

2015-12-08 Thread xguo27
Github user xguo27 commented on the pull request: https://github.com/apache/spark/pull/9553#issuecomment-162964945 Hi @yhuai, do you think this is good to merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-12074] Avoid memory copy involving Byte...

2015-12-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10177 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12074] Avoid memory copy involving Byte...

2015-12-08 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10177#issuecomment-162963794 Thanks @tedyu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5423#issuecomment-162963733 **[Test build #47346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47346/consoleFull)** for PR 5423 at commit [`b0e25bd`](https://gith

[GitHub] spark pull request: [SPARK-7889] [CORE] HistoryServer to refresh c...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46989032 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-11315] [YARN] Add YARN extension servic...

2015-12-08 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-162963641 Now that it's dependency PR is in, I welcome comments and reviews on this. The latest change just adds a version counter to every entity publishing, so that when c

[GitHub] spark pull request: [SPARK-12074] Avoid memory copy involving Byte...

2015-12-08 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10177#issuecomment-162963319 LGTM. Merging to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-7889] [CORE] WiP HistoryServer to refre...

2015-12-08 Thread steveloughran
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/6935#issuecomment-162962369 FWIW, I've got the SPARK-1537 Yarn history provider hooked up to this in [a branch](https://github.com/steveloughran/spark/tree/history/SPARK-7889%2BSPARK-1537-prov

[GitHub] spark pull request: [SPARK-7889] [CORE] WiP HistoryServer to refre...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46988309 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-7889] [CORE] WiP HistoryServer to refre...

2015-12-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6935#discussion_r46988368 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala --- @@ -0,0 +1,648 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] spark pull request: [SPARK-11155][Web UI] Stage summary json shoul...

2015-12-08 Thread keypointt
Github user keypointt commented on the pull request: https://github.com/apache/spark/pull/10107#issuecomment-162961226 @squito yeah! my first code change, and I've learnt a lot from you :+1: Thank you so much Imran! --- If your project is set up for it, you can reply to this ema

[GitHub] spark pull request: [SPARK-11315] [YARN] WiP Add YARN extension se...

2015-12-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8744#issuecomment-162960692 **[Test build #47345 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47345/consoleFull)** for PR 8744 at commit [`5acaebb`](https://gith

[GitHub] spark pull request: [SPARK-11155][Web UI] Stage summary json shoul...

2015-12-08 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/10107#issuecomment-162960042 merged to master (I made some very tiny style changes). Thanks for working on this @keypointt! --- If your project is set up for it, you can reply to this email and ha

<    1   2   3   4   5   6   7   8   >