[GitHub] spark pull request: [MLLIB] SPARK-2311: Added additional GLMs (Poi...
Github user xwei-datageek commented on the pull request: https://github.com/apache/spark/pull/1237#issuecomment-47742326 Could one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2325] Utils.getLocalDir had better chec...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/1281 [SPARK-2325] Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly If the first directory of spark.local.dir is bad, application will exit with the exception: Exception in thread main java.io.IOException: Failed to create a temp directory (under /data1/sparkenv/local) after 10 attempts! at org.apache.spark.util.Utils$.createTempDir(Utils.scala:258) at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:154) at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127) at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31) at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48) at org.apache.spark.broadcast.BroadcastManager.init(BroadcastManager.scala:35) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218) at org.apache.spark.SparkContext.init(SparkContext.scala:202) at JobTaskJoin$.main(JobTaskJoin.scala:9) at JobTaskJoin.main(JobTaskJoin.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly. For example, spark.local.dir is /data1/sparkenv/local,/data2/sparkenv/local. The disk data1 is bad while the disk data2 is good, we could choose the data2 not data1. You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-2325 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1281.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1281 commit 08424ce408b5e1ee679d15e46ea5b08979511fae Author: yantangzhai tyz0...@163.com Date: 2014-07-02T06:55:39Z [SPARK-2325] Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2325] Utils.getLocalDir had better chec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1281#issuecomment-47743236 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1949. Servlet 2.5 vs 3.0 conflict in SBT...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/906#issuecomment-47743422 Hi, I encountered this kind of Servlet API conflict. When are you planning to merge this? Or is there something to do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/1282 fix spark.yarn.max.executor.failures explaination According to '''scala private val maxNumExecutorFailures = sparkConf.getInt(spark.yarn.max.executor.failures, sparkConf.getInt(spark.yarn.max.worker.failures, math.max(args.numExecutors * 2, 3))) default value should be numExecutors * 2, with minimum of 3, and it's same to the config `spark.yarn.max.worker.failures` You can merge this pull request into a Git repository by running: $ git pull https://github.com/CrazyJvm/spark yarn-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1282.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1282 commit a4b2e27b0c2d2345a60ba66943b219968465b48a Author: CrazyJvm crazy...@gmail.com Date: 2014-07-02T06:59:48Z fix configuration spark.yarn.max.executor.failures commit 2900d234c6ebb90a5c4601083ddf8d329a2ee99d Author: CrazyJvm crazy...@gmail.com Date: 2014-07-02T07:04:51Z fix style commit 211f1302aa6d57b07a7b2d3b7cd4ab21e6d50bbd Author: CrazyJvm crazy...@gmail.com Date: 2014-07-02T07:06:28Z fix html tag commit 86effa612d2ec9ae991b43e229c4ed266e6605a6 Author: CrazyJvm crazy...@gmail.com Date: 2014-07-02T07:15:08Z change expression commit c438aecdec8ce90cb839b7c9aa8260ff4d3c62ba Author: CrazyJvm crazy...@gmail.com Date: 2014-07-02T07:18:18Z fix style --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1282#issuecomment-47744848 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...
Github user avulanov commented on a diff in the pull request:
https://github.com/apache/spark/pull/1270#discussion_r1439
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.evaluation
+
+import org.apache.spark.Logging
+import org.apache.spark.rdd.RDD
+import org.apache.spark.SparkContext._
+
+/**
+ * Evaluator for multilabel classification.
+ * NB: type Double both for prediction and label is retained
+ * for compatibility with model.predict that returns Double
+ * and MLUtils.loadLibSVMFile that loads class labels as Double
+ *
+ * @param predictionAndLabels an RDD of (predictions, labels) pairs, both
are non-null sets.
+ */
+class MultilabelMetrics(predictionAndLabels:RDD[(Set[Double],
Set[Double])]) extends Logging{
+
+ private lazy val numDocs = predictionAndLabels.count
+
+ private lazy val numLabels = predictionAndLabels.flatMap{case(_, labels)
= labels}.distinct.count
+
+ /**
+ * Returns strict Accuracy
+ * (for equal sets of labels)
+ * @return strictAccuracy.
+ */
+ lazy val strictAccuracy = predictionAndLabels.filter{case(predictions,
labels) =
+predictions == labels}.count.toDouble / numDocs
+
+ /**
+ * Returns Accuracy
+ * @return Accuracy.
+ */
+ lazy val accuracy = predictionAndLabels.map{ case(predictions, labels) =
+labels.intersect(predictions).size.toDouble /
labels.union(predictions).size}.
+fold(0.0)(_ + _) / numDocs
+
--- End diff --
Thanks, done!
https://github.com/avulanov/spark/commit/79e847656d8f062fad6a4c26a1f9a31dc59bed9d.
Could you also review https://github.com/apache/spark/pull/1155 ? It is my
pull request for multiclass classification measures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1282#issuecomment-47747966 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: fix spark.yarn.max.executor.failures explainat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1282#issuecomment-47747968 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16292/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47748924 FYI I believe `:quit` is considered the correct way in scala 2.11. You can still do something like `exit` but now it forces the user to go for `sys.exit`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2311: Added additional GLMs (Poi...
Github user BaiGang commented on the pull request: https://github.com/apache/spark/pull/1237#issuecomment-47749336 One more thing. Per our discussion in the line note, let's change SimpleUpdater to SquaredL2Updater. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2342] Evaluation helper's output type d...
GitHub user yijieshen opened a pull request: https://github.com/apache/spark/pull/1283 [SPARK-2342] Evaluation helper's output type doesn't conform to input ty... The function cast doesn't conform to the intention of Those expressions are supposed to be in the same data type, and also the return type. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yijieshen/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1283.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1283 commit c7aaa4bf1bd9f20c36674861f479949fad2f9f57 Author: Yijie Shen henry.yijies...@gmail.com Date: 2014-07-02T08:24:25Z [SPARK-2342] Evaluation helper's output type doesn't conform to input type --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2342] Evaluation helper's output type d...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1283#issuecomment-47750123 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47755181 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47755160 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47755626 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47755627 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16293/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47757403 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47757414 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-47757519 @BaiGang Thanks! I'm implementing the decomposition of multiclass and multilabel problems to binary classification problems that can be solved with built-in MLLib classifiers. I use one-vs-one and one-vs-all approaches. As far as I understand, MultiBoost.MH is a C++ implementation of Adaboost.MH and the latter uses another kind of problem decomposition in addition to boosting. So, our efforts are complimentary. Lets stay in touch. Btw, I would be glad to benchmark your classifier with the classification tasks that I'm solving. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47757826 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47757835 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47757837 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16294/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47758209 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47758210 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16295/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...
Github user edrevo commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47758337 All feedback has been addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47764577 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47764564 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Delete the useless import
GitHub user XuTingjun opened a pull request: https://github.com/apache/spark/pull/1284 Delete the useless import import org.apache.spark.util.Utils is never used in HistoryServer.scala You can merge this pull request into a Git repository by running: $ git pull https://github.com/XuTingjun/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1284.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1284 commit ff4ec06adb63ccfd0a62c5f9181c169eceff5ffb Author: XuTingjun 1039320...@qq.com Date: 2014-07-02T12:02:49Z Delete the useless import import org.apache.spark.util.Utils is never used in HistoryServer.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Delete the useless import
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1284#issuecomment-47767599 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47772513 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47772514 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16296/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47774796 Hi all,I have resolved the conflict and merged with the master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK
Github user yangliuyu commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-47795986 @vrilleup about your performance test result for real matrices, what's the cpu usage rate for each executor? We run svd on a 205899 x 1000 sparse matrix, 1850566 non-zeros, (Spark standalone mode, run 2 worker instance on one machine, one executor for each worker, 16 cores total) it cost 18 minutes not including getting the U matrix, cpu usage is about 200%-400% per executor. Seems not a reasonable time cost even on small number of executors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1949. Servlet 2.5 vs 3.0 conflict in SBT...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/906#issuecomment-47800209 See #1271. Sean, maybe close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2340] Resolve History Server file paths...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1280#issuecomment-47800445 LGMT. I'd have made the style changes in a separate PR, since they make the change appear much larger than it actually is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [STREAMING] SPARK-2343: Fix QueueInputDStream ...
GitHub user mlaflamm opened a pull request: https://github.com/apache/spark/pull/1285 [STREAMING] SPARK-2343: Fix QueueInputDStream with oneAtATime false Fix QueueInputDStream which was not removing dequeued items when used with the oneAtATime flag disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mlaflamm/spark spark-2343 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1285 commit c51d02973f3b233776220a67c1a6ba571068a0a9 Author: Manuel Laflamme manuel.lafla...@gmail.com Date: 2014-07-02T16:51:43Z Fix QueueInputDStream with oneAtATime false Fix QueueInputDStream which was not removing dequeued items when used with the oneAtATime flag disabled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [STREAMING] SPARK-2343: Fix QueueInputDStream ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1285#issuecomment-47804570 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1949. Servlet 2.5 vs 3.0 conflict in SBT...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/906#issuecomment-47804679 Close this? I encountered after the #1271 was merged. I believe at least we need to exclude from Hive related dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47805034 Hi Ximo, Sorry for the delay. Many of the committers are busy running the Spark Summit these few days so a lot of PRs are backlog'd. They will probably be addressed later this week / early next week. On Wed, Jul 2, 2014 at 3:06 AM, Ximo Guanter notificati...@github.com wrote: All feedback has been addressed â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/1211#issuecomment-47758337. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1151#discussion_r14469131
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -135,13 +137,12 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
}
}
- protected lazy val query: Parser[LogicalPlan] = (
-select * (
-UNION ~ ALL ^^^ { (q1: LogicalPlan, q2: LogicalPlan) = Union(q1,
q2) } |
-UNION ~ opt(DISTINCT) ^^^ { (q1: LogicalPlan, q2: LogicalPlan) =
Distinct(Union(q1, q2)) }
- )
-| insert | cache
- )
+ protected lazy val query: Parser[LogicalPlan] =
+ select * (
+ UNION ~ ALL ^^^ { (q1: LogicalPlan, q2: LogicalPlan) = Union(q1,
q2)} |
+ EXCEPT ^^^ { (q1: LogicalPlan, q2: LogicalPlan) = Except(q1, q2)} |
+ UNION ~ opt(DISTINCT) ^^^ { (q1: LogicalPlan, q2: LogicalPlan) =
Distinct(Union(q1, q2))}
+ ) | insert
--- End diff --
You are dropping `| cache` here, which is why the test case is failing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...
Github user edrevo commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47805299 No problem. Thanks for the update! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1151#discussion_r14469249
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -205,3 +205,18 @@ object ExistingRdd {
case class ExistingRdd(output: Seq[Attribute], rdd: RDD[Row]) extends
LeafNode {
override def execute() = rdd
}
+
+/**
+ * :: DeveloperApi ::
+ * This operator support the substract function .
+ * Return an table with the elements from `left` that are not in `right`.
--- End diff --
I think this comment can just read: Returns an table with the elements
from `left` that are not in `right` using the built-in spark `subtract`
function Also please name this `Except` to match the logical node.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1949. Servlet 2.5 vs 3.0 conflict in SBT...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/906#issuecomment-47805366 Ah, ok. I though you just had missed that other PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1211 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47805797 Thanks! I've merged this into master and the 1.0 branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-47806147 Thanks! I've merged this into master and the 1.0 branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1268 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2287] [SQL] Make ScalaReflection be abl...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1226 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2287] [SQL] Make ScalaReflection be abl...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1226#issuecomment-47806562 Thanks! I've merged this into master and the 1.0 branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Add timestamps to block manager events.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/654#issuecomment-47809756 Just another rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Streaming programming guide typos
GitHub user cykl opened a pull request: https://github.com/apache/spark/pull/1286 Streaming programming guide typos Fix a bad Java code sample and a broken link in the streaming programming guide. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cykl/spark streaming-programming-guide-typos Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1286.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1286 commit 9d3c53541767b30f3fd8e86c5537778ecbbf2faa Author: Clément MATHIEU clem...@unportant.info Date: 2014-07-02T17:17:31Z Spark streaming requires at least two working threads (scala version was OK) commit b0908cba1e1730818714c3d396980f4b6b71fa75 Author: Clément MATHIEU clem...@unportant.info Date: 2014-07-02T17:35:17Z Fix broken URL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Streaming programming guide typos
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1286#issuecomment-47812047 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47813702 Jenkins, add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47813772 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47813863 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47813872 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47815078 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16297/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] [SPARK-2222] Add multiclass evaluation...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1155#issuecomment-47815076 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2308][MLLIB] Add Mini-Batch KMeans Clus...
Github user rnowling commented on the pull request: https://github.com/apache/spark/pull/1248#issuecomment-47818425 Sean, I updated the code to factor out common bits into a KMeansCommons file, using traits for both the objects and classes. I updated the KMeansMiniBatch tests so they are customized for the KMeansMiniBatch, don't duplicate testing of common code, and account for the stochastic nature by using an epsilon for the errors instead of directly comparing the floats. I also realized that I failed to implement a key part of the MiniBatch algorithm so that is now included. Please review again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Branch 0.8
GitHub user salexln opened a pull request: https://github.com/apache/spark/pull/1287 Branch 0.8 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/spark branch-0.8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1287 commit 75bd03e4d9a6605e96c6dd405ef09b4975495f61 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T00:08:25Z [maven-release-plugin] prepare release v0.8.0-incubating commit 387892353f1f8abcce25f762bcd438ffb39a4f0d Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T00:08:31Z [maven-release-plugin] prepare for next development iteration commit d5f0ec1add357d960835dd54ee35f268f1c2aaf0 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T02:28:56Z Revert [maven-release-plugin] prepare for next development iteration This reverts commit 387892353f1f8abcce25f762bcd438ffb39a4f0d. commit 4ed1921a7b5c15eba5fab96dd2cb774fc2ea4a03 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T02:29:05Z Revert [maven-release-plugin] prepare release v0.8.0-incubating This reverts commit 75bd03e4d9a6605e96c6dd405ef09b4975495f61. commit ce6b881db90f6cec76789f6f9d0e66480327dde4 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T02:29:15Z Revert Hard code scala version in pom files. This reverts commit a1e7e519653beae47cafccbb9e335bc74986c0dc. commit 487205f3a16b2bf4a0db7cd70b70a5fd4a047c74 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T02:59:32Z Revert Revert Hard code scala version in pom files. This reverts commit ce6b881db90f6cec76789f6f9d0e66480327dde4. commit d9e80d53f6535f022ca59e11ffd1d164c17548d9 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T03:11:34Z [maven-release-plugin] prepare release v0.8.0-incubating commit 34aa342138a06913787ab1cec7ee7b57a14f7cba Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-16T03:11:50Z [maven-release-plugin] prepare for next development iteration commit b4fef78d95e19e4ea13483b7c93a38fd291c6592 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T05:58:28Z Revert [maven-release-plugin] prepare for next development iteration This reverts commit 34aa342138a06913787ab1cec7ee7b57a14f7cba. commit fc48d482a6d878c1b9358a382cc0f68ca86a3dda Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T05:58:32Z Revert [maven-release-plugin] prepare release v0.8.0-incubating This reverts commit d9e80d53f6535f022ca59e11ffd1d164c17548d9. commit 721ea8cf2080ab77f50c148a5c5aa5bab8d1b429 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T05:30:54Z Make spark-kafka a provided dependency. Otherwise any third-party build that links against spark-streaming will break (even if they aren't using kafka). This also defines the local repository inside of the examples pom so that the examples build continues to work. commit 72f8dcd9ddbe2fff8c64ec4a56892cd2a7593867 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T06:17:05Z [maven-release-plugin] prepare release v0.8.0-incubating commit fc9626d1448adc3d41855f9254fd9919a4157ad3 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T06:17:35Z Revert [maven-release-plugin] prepare release v0.8.0-incubating This reverts commit 72f8dcd9ddbe2fff8c64ec4a56892cd2a7593867. commit 3b85a8558da2c87873c85f227a189e45bf16b65d Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T06:27:03Z [maven-release-plugin] prepare release v0.8.0-incubating commit c71efa1a99fe6ba29e70afe59a2c08061e3fadab Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-17T06:27:10Z [maven-release-plugin] prepare for next development iteration commit df0a40ccfe4b18d5ad25b21fbcf0d2355aa37379 Author: Patrick Wendell pwend...@gmail.com Date: 2013-09-24T18:34:25Z Bumping versions in branch-0.8 commit d5a8dbfb2495c42f609960c28915c30fbe5541a6 Author: Reynold Xin reyno...@gmail.com Date: 2013-09-22T22:06:48Z Merge pull request #928 from jerryshao/fairscheduler-refactor Refactor FairSchedulableBuilder (cherry picked from commit 834686b108ce31cbee531d89de6c6e80913448f4) Signed-off-by: Reynold Xin r...@apache.org commit f3c60c9c0ce1f0c94174ecb903519b5fc4c696cd Author: Reynold Xin reyno...@gmail.com Date: 2013-09-22T05:43:34Z Merge pull request #941 from ilikerps/master Add org.apache. prefix to packages in spark-class (cherry picked from commit f06f2da2cb2fd6bd25b57e5f5fd6f8e5d37ab1a3) Signed-off-by: Reynold Xin r...@apache.org commit a186792af072064bdaa65b94d80e394ec911302c Author: Reynold Xin reyno...@gmail.com Date:
[GitHub] spark pull request: Branch 0.8
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1287#issuecomment-47820349 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Branch 0.8
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1287#issuecomment-47820592 Do you mind closing this pull request? I don't think it does what you wanted it go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Branch 0.8
Github user salexln closed the pull request at: https://github.com/apache/spark/pull/1287 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1949. Servlet 2.5 vs 3.0 conflict in SBT...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/906#issuecomment-47836900 Yeah @vanzin these are actually different changes. Incarnations of the same general issue. @pwendell if you have a moment to have a second look, I think this actually works now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK] Fix NPE for ExternalAppendOnlyMap
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/1288 [SPARK] Fix NPE for ExternalAppendOnlyMap It did not handle null keys very gracefully before. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewor14/spark fix-external Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1288.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1288 commit ed5adf954f5edbcf951a3809f0dc945c94c92d26 Author: Andrew Or andrewo...@gmail.com Date: 2014-07-02T21:42:55Z Fix NPE for ExternalAppendOnlyMap commit 312b8d805097127ba763a201fd480b7fef089519 Author: Andrew Or andrewo...@gmail.com Date: 2014-07-02T23:11:16Z Abstract key hash code --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK] Fix NPE for ExternalAppendOnlyMap
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1288#issuecomment-47849204 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK] Fix NPE for ExternalAppendOnlyMap
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1288#issuecomment-47849216 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK] Fix NPE for ExternalAppendOnlyMap
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1288#issuecomment-47852272 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16298/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK] Fix NPE for ExternalAppendOnlyMap
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1288#issuecomment-47852270 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2350] Don't NPE while launching drivers
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/1289 [SPARK-2350] Don't NPE while launching drivers Prior to this change, we could throw a NPE if we launch a driver while another one is waiting, because removing from an iterator while iterating over it is not safe. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aarondav/spark master-fail Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1289 commit 1cf1cf48aec38ab971919533775ae8b5b70bfeef Author: Aaron Davidson aa...@databricks.com Date: 2014-07-03T00:14:52Z SPARK-2350: Don't NPE while launching drivers Prior to this change, we could throw a NPE if we launch a driver while another one is waiting, because removing from an iterator while iterating over it is not safe. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2350] Don't NPE while launching drivers
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1289#issuecomment-47853543 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2350] Don't NPE while launching drivers
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1289#issuecomment-47853550 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2350] Don't NPE while launching drivers
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1289#issuecomment-47856096 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16299/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2350] Don't NPE while launching drivers
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1289#issuecomment-47856095 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47856867 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47856855 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47856954 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16300/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47856953 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47857426 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47857418 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47863416 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47863417 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16301/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Delete the useless import
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1284#issuecomment-47864197 Hi @XuTingjun I took care of these in #1280 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47864285 Thanks a lot, Michael,I have modify the code . Merge build use two hours .But I saw the console test log error. I don't know if the new code is the main cause or the code could be merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...
GitHub user bgreeven opened a pull request: https://github.com/apache/spark/pull/1290 [spark-2352] Implementation of an 1-hidden layer Artificial Neural Network (ANN) The code contains a single layer ANN, with variable number of inputs, outputs and hidden nodes. It takes as input an RDD vector pairs, corresponding to the training set with inputs and outputs. A test program is also included, which also contains a graphical representation that can be switched on using the graph parameter. Without it, the summed squared error over the testing set is displayed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bgreeven/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1290 commit e60d8786d16e55264a61d8afc8d67bf06068aaaf Author: Bert Greevenbosch bert.greevenbo...@huawei.com Date: 2014-07-03T03:33:22Z Create ParallelANN.scala This is the main ParallelANN class and associated Model commit 52da23d54254b38eea7181872e4caa25981c028e Author: Bert Greevenbosch bert.greevenbo...@huawei.com Date: 2014-07-03T03:34:41Z Create GeneralizedSteepestDescendAlgorithm This is the general steepest descend model, with as inputs Vectors and outputs Vectors or Doubles. commit 152b8baf84742ecd0c622d41d3804eb74c0310a3 Author: Bert Greevenbosch bert.greevenbo...@huawei.com Date: 2014-07-03T03:36:31Z Create TestParallelANN.scala This is a test program for parallel ANNs. commit c8af840149ac8d1903afe4ac826a626d030bd385 Author: Bert Greevenbosch bert.greevenbo...@huawei.com Date: 2014-07-03T03:42:29Z Create TestParallelANNgraphics.scala Visualisation tools; only used when TestParallelANN is given the graph parameter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [spark-2352] Implementation of an 1-hidden lay...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-47864843 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47865791 modify according @cloud-fan âs comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866266 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: FIX: ShuffledDStream run tasks only when dstre...
GitHub user guowei2 opened a pull request: https://github.com/apache/spark/pull/1291 FIX: ShuffledDStream run tasks only when dstream has partition items ShuffledDStream run tasks no matter whether dstream has partition items this patch fixed it You can merge this pull request into a Git repository by running: $ git pull https://github.com/guowei2/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 749b632d0a53487a440f04051152eef072a32e15 Author: guowei guo...@upyoo.com Date: 2014-07-03T04:38:24Z SITUATION: ShuffledDStream run tasks whether dstream has partition items or not FIX: ShuffledDStream run tasks only when dstream has partition items --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866389 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: FIX: ShuffledDStream run tasks only when dstre...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1291#issuecomment-47866380 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866381 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866583 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Update SQLConf.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866584 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16302/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2235][SQL]Spark SQL basicOperator add I...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1150#issuecomment-47869343 Hi all, I have resolved the conflict . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2237][CORE]Add ZLIBCompressionCodec cod...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1121#issuecomment-47869431 Hi all,I have resolve the conflict i don't know if this pr has value to be merged? Thanks a lot --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1127#issuecomment-47869475 Hi all ,I have resolve the conflict . I don't know if this pr has the value to be merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
