[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67124588 @JoshRosen Thanks for the PR! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67124660 [Test build #24488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24488/consoleFull) for PR 2872 at commit [`be9cb43`](https://github.com/apache/spark/commit/be9cb43e48637d45b8a2936b2cdfb1971bf556a0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67124649 [Test build #24487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24487/consoleFull) for PR 3301 at commit [`1d7422f`](https://github.com/apache/spark/commit/1d7422fe07dc4a39448334dc1ca1c1a5c702f935). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67124723 Wow, this ended up being a net removal of code, even with the extra docs + generalization. Nice! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67125228 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67125457 [Test build #24489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24489/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3955] Different versions between jackso...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/2818#issuecomment-67125929 I close this PR because of fixing it via #3379 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3955] Different versions between jackso...
Github user jongyoul closed the pull request at: https://github.com/apache/spark/pull/2818 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user suyanNone closed the pull request at: https://github.com/apache/spark/pull/3629 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67126072 It is already resolved in [SPARK-3000][CORE] drop old blocks to disk in parallel when free memory is not enough for caching new blocks #2134 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3000][CORE] drop old blocks to disk in ...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/2134#issuecomment-67126094 yes, its duplicate with your patch I just see you patch title parallel drop to disk... so I don't see the code in detail. I already close my patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67126547 Hi @suyanNone , you don't need to close this PR, since [#2134](https://github.com/apache/spark/pull/2134) is not merged yet. And the bug still exists in current code. You can reopen this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67126652 [Test build #24485 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24485/consoleFull) for PR 3301 at commit [`95d13eb`](https://github.com/apache/spark/commit/95d13eb335f441ba34615bada691655702df63a1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67126659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24485/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67127181 [Test build #24490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24490/consoleFull) for PR 3710 at commit [`f5285d6`](https://github.com/apache/spark/commit/f5285d6628185c2bf21e3d1e3d7f72803cc515a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67127309 Reopen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67127268 @liyezhang556520 I not familiar with the process about pull request = =, ok, I will reopen it... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
GitHub user suyanNone reopened a pull request: https://github.com/apache/spark/pull/3629 [SPARK-4777][CORE] Some block memory after unrollSafely not count into used memory(memoryStore.entrys or unrollMemory) Some memory not count into memory used by memoryStore or unrollMemory. Thread A after unrollsafely memory, it will release 40MB unrollMemory(40MB will used by other threads). then ThreadA wait get accountingLock to tryToPut blockA(30MB). before Thread A get accountingLock, blockA memory size is not counting into unrollMemory or memoryStore.currentMemory. IIUC, freeMemory should minus that block memory So, put this release memory into pending, and release it in tryToPut before ensureSpace You can merge this pull request into a Git repository by running: $ git pull https://github.com/suyanNone/spark unroll-memory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3629.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3629 commit 072e43d49226f1ae660d9b2ad53dc43ee78481e9 Author: hushan[è¡ç] hus...@xiaomi.com Date: 2014-12-05T02:56:20Z Pending unroll memory for this block untill tryToPut commit 4f46dd7dc985d6e9eeb907f3f3e9ab010fa2f27d Author: hushan[è¡ç] hus...@xiaomi.com Date: 2014-12-08T07:15:58Z Make sure not add pending in every releaseUnrollMemory call commit 42dfa9ba9b092ad935740d9ca75a0a8166474d62 Author: hushan[è¡ç] hus...@xiaomi.com Date: 2014-12-11T02:28:37Z Refine getOrElse commit 34cfbe8b309addb98deb23429626b14cb13a8e2a Author: hushan[è¡ç] hus...@xiaomi.com Date: 2014-12-11T07:08:41Z Refine blockManagerSuite unroll test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67127610 [Test build #24491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24491/consoleFull) for PR 3629 at commit [`34cfbe8`](https://github.com/apache/spark/commit/34cfbe8b309addb98deb23429626b14cb13a8e2a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4516] Avoid allocating Netty PooledByte...
Github user liuxinglanyue commented on the pull request: https://github.com/apache/spark/pull/3465#issuecomment-67128796 thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67129497 [Test build #24492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24492/consoleFull) for PR 3319 at commit [`065b531`](https://github.com/apache/spark/commit/065b53181349fa0cc56d4828044b1d564791ea80). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add unit test to spark_ec2 script
Github user logc closed the pull request at: https://github.com/apache/spark/pull/134 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67130002 [Test build #24493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24493/consoleFull) for PR 3319 at commit [`d8be7bc`](https://github.com/apache/spark/commit/d8be7bc07b23982c4fced647f85982c6b7cadd4b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-67130169 I pushed some more changes (this is still WIP). In addition to our use of Thread.sleep, I think I should also look out for cases where streaming directly calls System.currentTimeMillis or System.nanoTime instead of `Clock.currentTime`, since this could be another source of time races. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
GitHub user ksakellis opened a pull request: https://github.com/apache/spark/pull/3711 [SPARK-4857] [CORE] Adds Executor membership events to SparkListener Adds onExecutorAdded and onExecutorRemoved events to the SparkListener. This will allow a client to get notified when an executor has been added/removed and provide additional information such as how many vcores it is consuming. In addition, this commit adds a SparkListenerAdapter to the Java API that provides default implementations to the SparkListener. This is to get around the fact that default implementations for traits don't work in Java. Having Java clients extend SparkListenerAdapter moving forward will prevent breakage in java when we add new events to SparkListener. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ksakellis/spark kostas-spark-4857 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3711 commit b1f715d2c4c169bbc33175ea7984af22a7b941f8 Author: Kostas Sakellis kos...@cloudera.com Date: 2014-12-16T08:55:01Z [SPARK-4857] [CORE] Adds Executor membership events to SparkListener Adds onExecutorAdded and onExecutorRemoved events to the SparkListener. This will allow a client to get notified when an executor has been added/removed and provide additional information such as how many vcores it is consuming. In addition, this commit adds a SparkListenerAdapter to the Java API that provides default implementations to the SparkListener. This is to get around the fact that default implementations for traits don't work in Java. Having Java clients extend SparkListenerAdapter moving forward will prevent breakage in java when we add new events to SparkListener. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884615
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
---
@@ -74,12 +74,14 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F :
NewInputFormat[K,V] : Clas
newFilesOnly: Boolean = true)
extends InputDStream[(K, V)](ssc_) {
+ @transient private val clock = ssc.scheduler.clock
--- End diff --
The changes in this file are just to allow the clock to be mocked.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884670
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -305,7 +305,7 @@ class CheckpointSuite extends TestSuiteBase {
// var clock = ssc.scheduler.clock.asInstanceOf[ManualClock]
Thread.sleep(1000)
for (i - Seq(1, 2, 3)) {
- Files.write(i + \n, new File(testDir, i.toString),
Charset.forName(UTF-8))
+ Files.write(i + \n, new File(testDir, i.toString), Charsets.UTF_8)
--- End diff --
As long as I was already modifying this file, I figured I could clean this
up to match our convention from #2781.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-67130485 [Test build #24495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24495/consoleFull) for PR 3687 at commit [`ee8c8f8`](https://github.com/apache/spark/commit/ee8c8f8013560345e3da1d808bc142c05a1ce70b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67130479 [Test build #24494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24494/consoleFull) for PR 3711 at commit [`b1f715d`](https://github.com/apache/spark/commit/b1f715d2c4c169bbc33175ea7984af22a7b941f8). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class SparkListenerAdapter implements SparkListener ` * `case class SparkListenerExecutorAdded(executorId: String, executorInfo : ExecutorInfo)` * `case class SparkListenerExecutorRemoved(executorId: String, executorInfo : ExecutorInfo)` * `class ExecutorInfo(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67130482 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24494/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67130472 [Test build #24494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24494/consoleFull) for PR 3711 at commit [`b1f715d`](https://github.com/apache/spark/commit/b1f715d2c4c169bbc33175ea7984af22a7b941f8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884849
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -238,14 +190,17 @@ class InputStreamsSuite extends TestSuiteBase with
BeforeAndAfter {
val testDir: File = null
try {
val testDir = Utils.createTempDir()
+ // Create a file that exists before the StreamingContext is created
val existingFile = new File(testDir, 0)
- Files.write(0\n, existingFile, Charset.forName(UTF-8))
+ Files.write(0\n, existingFile, Charsets.UTF_8)
+ assert(existingFile.setLastModified(1))
- Thread.sleep(1000)
// Set up the streaming context and input streams
- val newConf = conf.clone.set(
-spark.streaming.clock,
org.apache.spark.streaming.util.SystemClock)
--- End diff --
This usage of `SystemClock` was a little tricky to fix. It looks like the
reason that this test used `Thread.sleep()` was that `FileInputDStream` relied
on the filesystem timestamps matching the system clock when determining whether
files were new. Instead, we can just manually those files' modification times.
I had to spend a bit of time fiddling with the actual constants used in this
test, so it would be great if someone could take a look to make sure I haven't
inadvertently broken the test's ability to catch bugs (might be good to
whiteboard out a reasonable set of constants in terms of batchInterval, window
size, etc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4797] Replace breezeSquaredDistance
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/3643#discussion_r21885069
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala ---
@@ -264,6 +263,92 @@ object MLUtils {
}
Vectors.fromBreeze(vector1)
}
+
+ /**
+ * Returns the squared distance between two Vectors.
+ */
+ private[util] def vectorSquaredDistance(v1: Vector, v2: Vector): Double
= {
+var squaredDistance = 0.0
+(v1, v2) match {
+ case (v1: SparseVector, v2: SparseVector) =
+val v1Values = v1.values
+val v1Indices = v1.indices
+val v2Values = v2.values
+val v2Indices = v2.indices
+val nnzv1 = v1Indices.size
+val nnzv2 = v2Indices.size
+
+var kv1 = 0
+var kv2 = 0
+var score = 0.0
+while (kv1 nnzv1) {
+ val iv1 = v1Indices(kv1)
+
+ if (kv2 = nnzv2 || iv1 v2Indices(kv2)) {
+score = v1Values(kv1)
+squaredDistance += score * score
+ }
+ while (kv2 nnzv2 v2Indices(kv2) iv1) {
+score = v2Values(kv2)
+squaredDistance += score * score
+kv2 += 1
+ }
+ if (kv2 nnzv2 v2Indices(kv2) == iv1) {
+score = v1Values(kv1) - v2Values(kv2)
+squaredDistance += score * score
+kv2 += 1
+ }
+ kv1 += 1
+}
+
+ case (v1: SparseVector, v2: DenseVector) if v1.indices.length /
v1.size 0.5 =
+squaredDistance = vectorSquaredDistance(v1, v2)
+
+ case (v1: DenseVector, v2: SparseVector) if v2.indices.length /
v2.size 0.5 =
+squaredDistance = vectorSquaredDistance(v2, v1)
+
+ case (v1, v2) =
+squaredDistance =
v1.toArray.zip(v2.toArray).foldLeft(0.0)((distance, elems) = {
+ val score = elems._1 - elems._2
+ distance + score * score
+})
+}
+squaredDistance
+ }
+
+ /**
+ * Returns the squared distance between DenseVector and SparseVector.
+ */
+ private[util] def vectorSquaredDistance(v1: SparseVector, v2:
DenseVector): Double = {
--- End diff --
Indeed. Modified in later commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67132172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24488/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2872#issuecomment-67132162 [Test build #24488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24488/consoleFull) for PR 2872 at commit [`be9cb43`](https://github.com/apache/spark/commit/be9cb43e48637d45b8a2936b2cdfb1971bf556a0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67132292 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24487/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67132282 [Test build #24487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24487/consoleFull) for PR 3301 at commit [`1d7422f`](https://github.com/apache/spark/commit/1d7422fe07dc4a39448334dc1ca1c1a5c702f935). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67133181 [Test build #24489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24489/consoleFull) for PR 3607 at commit [`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3607#issuecomment-67133187 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24489/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67133566 [Test build #24496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24496/consoleFull) for PR 3711 at commit [`ab2575f`](https://github.com/apache/spark/commit/ab2575fa917c59c3ae7c18342dccf009507e22a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67133754 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24491/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4777][CORE] Some block memory after unr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3629#issuecomment-67133743 [Test build #24491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24491/consoleFull) for PR 3629 at commit [`34cfbe8`](https://github.com/apache/spark/commit/34cfbe8b309addb98deb23429626b14cb13a8e2a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67135310 [Test build #24490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24490/consoleFull) for PR 3710 at commit [`f5285d6`](https://github.com/apache/spark/commit/f5285d6628185c2bf21e3d1e3d7f72803cc515a4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4859][Streaming] Improve StreamingListe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3710#issuecomment-67135315 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24490/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67136033 [Test build #24492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24492/consoleFull) for PR 3319 at commit [`065b531`](https://github.com/apache/spark/commit/065b53181349fa0cc56d4828044b1d564791ea80). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67136043 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24492/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67136740 [Test build #24493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24493/consoleFull) for PR 3319 at commit [`d8be7bc`](https://github.com/apache/spark/commit/d8be7bc07b23982c4fced647f85982c6b7cadd4b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67136754 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24493/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4797] Replace breezeSquaredDistance
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/3643#issuecomment-67137308 @jkbradley Thanks. The codes are modified for your comments. The test is also expanded to test the case of the major comment you mentioned. Please check it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-67137409 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24495/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3687#issuecomment-67137400 [Test build #24495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24495/consoleFull) for PR 3687 at commit [`ee8c8f8`](https://github.com/apache/spark/commit/ee8c8f8013560345e3da1d808bc142c05a1ce70b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67141113 [Test build #24497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24497/consoleFull) for PR 3319 at commit [`65c562e`](https://github.com/apache/spark/commit/65c562e57078ccb31de281b238a9348dd9a1f7c2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67143159 [Test build #24496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24496/consoleFull) for PR 3711 at commit [`ab2575f`](https://github.com/apache/spark/commit/ab2575fa917c59c3ae7c18342dccf009507e22a4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class SparkListenerAdapter implements SparkListener ` * `case class SparkListenerExecutorAdded(executorId: String, executorInfo : ExecutorInfo)` * `case class SparkListenerExecutorRemoved(executorId: String, executorInfo : ExecutorInfo)` * `class ExecutorInfo(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4857] [CORE] Adds Executor membership e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3711#issuecomment-67143165 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24496/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21892548 --- Diff: core/pom.xml --- @@ -352,9 +352,9 @@ /execution /executions configuration - tasks + target --- End diff -- Yes, Ant reports that `tasks` is deprecated: https://gist.github.com/srowen/ddf5e606ba9cb888999f#file-gistfile1-txt-L13 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3157#discussion_r21892585
--- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java ---
@@ -1556,7 +1558,7 @@ public void testGuavaOptional() {
@Test
public void testRegisterKryoClasses() {
SparkConf conf = new SparkConf();
-conf.registerKryoClasses(new Class[]{ Class1.class, Class2.class });
+conf.registerKryoClasses(new Class?[]{ Class1.class, Class2.class });
--- End diff --
It's the general uses unchecked or unsafe operations warning:
https://gist.github.com/srowen/ddf5e606ba9cb888999f#file-gistfile1-txt-L36
`Class` definitely carries a generic type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3319#issuecomment-67149415 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24497/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3157#discussion_r21893546
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala ---
@@ -450,7 +452,9 @@ class ParquetQuerySuite extends QueryTest with
FunSuiteLike with BeforeAndAfterA
}
test(make RecordFilter for simple predicates) {
-def checkFilter[T : FilterPredicate](predicate: Expression, defined:
Boolean = true): Unit = {
+def checkFilter[T : FilterPredicate : ClassTag](
--- End diff --
Oops, pushed one more commit that really fixes it. I think that's the last
of the ones you highlighted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67151644 [Test build #24498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24498/consoleFull) for PR 3157 at commit [`15e687d`](https://github.com/apache/spark/commit/15e687d9258b120c3f8b0938ea1c28120e9a8d8c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4547 [MLLIB] [WIP] OOM when making bins ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3702#issuecomment-67152810 @jkbradley Yes let's do `numBins`, I'm changing it now. Yeah, say you have 100 elements in 10 partitions, and want to sample down to 12. That means sampling about every 100/12 ~= 8th element. But the simplistic approach samples 20 elements, since each of 10 partitions will squash 1-8 and 9-10 into 2 new elements. Ideally 9-10 belong with 1-6 of the next partition or something. But stitching that together seems like more trouble than it's worth, or am I being pessimistic/lazy? or maybe I misunderstand your idea of offsets into the partition. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4156 [MLLIB] EM algorithm for GMMs
Github user tgaloppo commented on the pull request: https://github.com/apache/spark/pull/3022#issuecomment-67158536 I've merged in the predict() method from @FlytxtRnD I am working on the changeover from accumulators to RDD.aggregate; I should have this up soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...
Github user tdhopper commented on the pull request: https://github.com/apache/spark/pull/2895#issuecomment-67162827 @JoshRosen: Yup. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...
Github user tdhopper closed the pull request at: https://github.com/apache/spark/pull/2895 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67173661 [Test build #24499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24499/consoleFull) for PR 3157 at commit [`9556a83`](https://github.com/apache/spark/commit/9556a83a96eab3c3894d8d7abc178631fb62a5f2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4861][SQL] Refactory command in spark s...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/3712 [SPARK-4861][SQL] Refactory command in spark sql Remove ```Command``` and use ```RunnableCommand``` instead. You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark cmd Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3712.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3712 commit 125f542da6d8f7d79646ed26bc805ddcfb5081a8 Author: scwf wangf...@huawei.com Date: 2014-12-16T13:21:55Z factory command in spark sql --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4861][SQL] Refactory command in spark s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3712#issuecomment-67178751 [Test build #24500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24500/consoleFull) for PR 3712 at commit [`125f542`](https://github.com/apache/spark/commit/125f542da6d8f7d79646ed26bc805ddcfb5081a8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4861][SQL] Refactory command in spark s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3712#issuecomment-67185695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24500/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4861][SQL] Refactory command in spark s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3712#issuecomment-67185687 [Test build #24500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24500/consoleFull) for PR 3712 at commit [`125f542`](https://github.com/apache/spark/commit/125f542da6d8f7d79646ed26bc805ddcfb5081a8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SetCommand(kv: Option[(String, Option[String])], _output: Seq[Attribute])(` * `case class UncacheTableCommand(tableName: String) extends RunnableCommand ` * `case class DescribeCommand(child: SparkPlan, _output: Seq[Attribute])(` * `case class AnalyzeTable(tableName: String)(hiveContext: HiveContext) extends RunnableCommand ` * `case class DropTable(` * `case class AddJar(path: String)(hiveContext: HiveContext) extends RunnableCommand ` * `case class AddFile(path: String)(hiveContext: HiveContext) extends RunnableCommand ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67193117 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24499/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67193105 [Test build #24499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24499/consoleFull) for PR 3157 at commit [`9556a83`](https://github.com/apache/spark/commit/9556a83a96eab3c3894d8d7abc178631fb62a5f2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4839][YARN] Adding a documentation abou...
Github user oza commented on the pull request: https://github.com/apache/spark/pull/3689#issuecomment-67195311 Updated a description of dynamic resource allocation based on a comment by @ash211. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4501][Core] - Create build/mvn to autom...
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/3707#issuecomment-67205414 @ScrapCodes good suggestion on the jenkins output. Modified it to print a single echo statement if a download is needed and quieted or silenced the `curl` / `wget` output. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4501][Core] - Create build/mvn to autom...
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/3707#issuecomment-67207337 @pwendell how is this looking from your end given the previous discussions through email threads? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4079] [CORE] Consolidates Errors if a C...
Github user ksakellis commented on a diff in the pull request:
https://github.com/apache/spark/pull/3119#discussion_r21922602
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -120,6 +129,12 @@ class LZFCompressionCodec(conf: SparkConf) extends
CompressionCodec {
@DeveloperApi
class SnappyCompressionCodec(conf: SparkConf) extends CompressionCodec {
+ try {
+Snappy.getNativeLibraryVersion
--- End diff --
@pwendell can you point to the api you are talking about?
https://github.com/xerial/snappy-java/tree/1.1.1.6/src/main/java/org/xerial/snappy
I don't see a Snappy.isLoaded() API. There is a member variable in
SnappyLoader but that is private.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4092] [CORE] Fix InputMetrics for coale...
Github user ksakellis commented on the pull request: https://github.com/apache/spark/pull/3120#issuecomment-67213648 @kayousterhout The test you pointed out: sc.parallelize(1 to 2).saveAsTextFile(file:tester1) val a = sc.textFile(file:tester1) sc.parallelize(1 to 10).saveAsTextFile(file:tester2) val b = sc.textFile(file:tester2) a.cartesian(b) b.cartesian(a) is actually not valid because of how cartesian is implemented. I added a comment to the interleaved reads unit test to describe the reasoning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3301#issuecomment-67215318 Alright, this looks good to me so I'll merge it into `master`. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67215577 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4437] update doc for WholeCombineFileRe...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3301 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4298][Core] - The spark-submit cannot r...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3561#issuecomment-67215790 [Test build #24501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24501/consoleFull) for PR 3561 at commit [`14daa20`](https://github.com/apache/spark/commit/14daa202030f8da4908b7dbce46b94cd8116d2c4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/3157#discussion_r21924054
--- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java ---
@@ -1556,7 +1558,7 @@ public void testGuavaOptional() {
@Test
public void testRegisterKryoClasses() {
SparkConf conf = new SparkConf();
-conf.registerKryoClasses(new Class[]{ Class1.class, Class2.class });
+conf.registerKryoClasses(new Class?[]{ Class1.class, Class2.class });
--- End diff --
That warning does not show up for me. Do you need some higher-than-default
warning setting turned on for it? (The fix does seem fine; I'm just wondering.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4595][Core] Fix MetricsServlet not work...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3444#issuecomment-67217026 I'm finally getting around to reviewing this now. Is there a good way to test this? What's the error message / symptom thatI should look for in the buggy version of this code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-67217419 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-67217396 [Test build #24502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24502/consoleFull) for PR 3696 at commit [`c6f9395`](https://github.com/apache/spark/commit/c6f93958a0147f56ed350dffad4f4e302a53687d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-67218182 [Test build #24503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24503/consoleFull) for PR 3688 at commit [`ebd2007`](https://github.com/apache/spark/commit/ebd2007d82ea909b1d060114148e4abc4f749b25). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4855][mllib] testing the Chi-squared hy...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3679#issuecomment-67218225 Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4855][mllib] testing the Chi-squared hy...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3679 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/3157#discussion_r21925414
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala ---
@@ -450,7 +452,9 @@ class ParquetQuerySuite extends QueryTest with
FunSuiteLike with BeforeAndAfterA
}
test(make RecordFilter for simple predicates) {
-def checkFilter[T : FilterPredicate](predicate: Expression, defined:
Boolean = true): Unit = {
+def checkFilter[T : FilterPredicate : ClassTag](
--- End diff --
verified!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4409][MLlib] Additional Linear Algebra ...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/3319#discussion_r21925528
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
---
@@ -313,15 +593,145 @@ object Matrices {
* @return Square `DenseMatrix` with size `values.length` x
`values.length` and `values`
* on the diagonal
*/
- def diag(vector: Vector): Matrix = {
-val n = vector.size
-val matrix = Matrices.eye(n)
-val values = vector.toArray
-var i = 0
-while (i n) {
- matrix.update(i, i, values(i))
- i += 1
+ def diag(vector: Vector): Matrix = DenseMatrix.diag(vector)
+
+ /**
+ * Horizontally concatenate a sequence of matrices. The returned matrix
will be in the format
+ * the matrices are supplied in. Supplying a mix of dense and sparse
matrices will result in
+ * a dense matrix.
--- End diff --
I like the MATLAB approach better. Usually a sparse matrix is very sparse,
while a dense component is quite small, for example,
~~~
A^T A A^T
A I
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/3157#discussion_r21925735 --- Diff: core/pom.xml --- @@ -352,9 +352,9 @@ /execution /executions configuration - tasks + target --- End diff -- Interesting, I don't see this. I also don't get any warnings prefaced with WARNING (in caps). I wonder if it's because I'm running via sbt instead of using maven directly, or something like that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/3157#issuecomment-67219288 @srowen LGTM The few items I couldn't verify myself look reasonable to me, so I'd say it's ready to go. CC: @pwendell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4839][YARN] Adding a documentation abou...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3689#issuecomment-67219777 Hey @oza thanks for doing this. I actually intend to structure this a little differently in its own separate page and there is another JIRA (SPARK-4140) that I filed earlier for this. I'll take this up and work on it in the next day or two. In the mean time would you mind closing this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4595][Core] Fix MetricsServlet not work...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3444#issuecomment-67220069 Ah, this was actually a pretty nasty bug since it seems to silently fail! I'm fine merging this patch to fix this issue, but I wonder whether there are other changes that we could make to `MetricsSystem` in order to prevent these sorts of issues. For example, it looks like `MetricsSystem` has a bunch of public methods like `getServletHandlers` that don't indicate when they're safe to call (e.g. that they are only safe to call after the `MetricsSystem` has been started). Similarly, there are a bunch of public methods like `registerSources`, `buildRegistryName`, and `registerSinks()` that should be private. I'd be happy to do this cleanup here, but maybe we should just merge this as-is then do the cleanup in a subsequent PR. Actually, let's try this: I'll submit a PR to your PR which does the cleanup in `MetricsSystem.scala` in order to prevent these errors from occurring silently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user judynash commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67220291 Thanks Cheng and Josh for restarting the tests! What's the next step now the test has passed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3675][SQL] Allow starting a JDBC server...
Github user chernetsov commented on the pull request: https://github.com/apache/spark/pull/2515#issuecomment-67220789 @marmbrus Hi, Michael. I am trying to make use of the commit you introduced (access my rdd exposed to sqlcontext via registerTempTable) but have problems with it: - I cherry-picked the change to 1.1.0 release - edited my spark app to call `HiveThriftServer2.startWithContext(sqlContext)` and added app thread wait() in the very end so that job does not finish - launched my app via spark-submit on local master - all the programmatical sql operations (like `sqlContext.sql(select * from ...).collect().foreach(println)`) work fine - thift server starts as well - but connecting via beeline shows that no tables are exposed to it ``` 0: jdbc:hive2://localhost:1 show tables; +-+ | result | +-+ +-+ No rows selected (0.136 seconds) ``` Is there any other trick I should do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-67220945 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-67221307 [Test build #24504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24504/consoleFull) for PR 3409 at commit [`daec3d0`](https://github.com/apache/spark/commit/daec3d01c937d80961b0f9eec4e0ad96539bd421). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3675][SQL] Allow starting a JDBC server...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2515#issuecomment-67221570 Temporary tables don't show up in `show tables` (we should fix this, feel free to open a JIRA). You should still be able to query it though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4595][Core] Fix MetricsServlet not work...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3444#issuecomment-67222433 I've opened a PR against this PR in order to add defensive checks to MetricsSystem, which would have prevented the issue fixed by this PR: https://github.com/jerryshao/apache-spark/pull/10 @jerryshao would be great if you could take a look at that and merge it into this PR branch if it looks fine to you (I've opened it against this branch). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4864] Add documentation to Netty-based ...
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/3713 [SPARK-4864] Add documentation to Netty-based configs You can merge this pull request into a Git repository by running: $ git pull https://github.com/aarondav/spark netty-configs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3713 commit 3b1f84e8f2f6aa736f3c625e42926cb1e0c25381 Author: Aaron Davidson aa...@databricks.com Date: 2014-12-16T19:53:05Z [SPARK-4864] Add documentation to Netty-based configs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-67222875 I don't really know anything about the thriftserver, so I'm not comfortable merging this PR myself. Therefore, let's ping @marmbrus for a final sign-off + commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
