[jira] [Commented] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043079#comment-14043079 ] Marcelo Vanzin commented on SPARK-2268: --- I'll look more closely, but we have users

[jira] [Created] (SPARK-2273) online learning algorithms: Passive Aggressive

Jun Xie created SPARK-2273: -- Summary: online learning algorithms: Passive Aggressive Key: SPARK-2273 URL: https://issues.apache.org/jira/browse/SPARK-2273 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-2273) online learning algorithms: Passive Aggressive

[ https://issues.apache.org/jira/browse/SPARK-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043083#comment-14043083 ] Jun Xie commented on SPARK-2273: I'd like to have this assigned to me because I

[jira] [Commented] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043088#comment-14043088 ] Mridul Muralidharan commented on SPARK-2268: That is not because of this hook.

[jira] [Commented] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043102#comment-14043102 ] Sean Owen commented on SPARK-2268: -- Yeah this hook is deleting local files, not HDFS

[jira] [Created] (SPARK-2274) spark SQL query hang up sometime

jackielihf created SPARK-2274: - Summary: spark SQL query hang up sometime Key: SPARK-2274 URL: https://issues.apache.org/jira/browse/SPARK-2274 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-2274) spark SQL query hang up sometimes

[ https://issues.apache.org/jira/browse/SPARK-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackielihf updated SPARK-2274: -- Summary: spark SQL query hang up sometimes (was: spark SQL query hang up sometime) spark SQL query

[jira] [Resolved] (SPARK-2267) Log exception when TaskResultGetter fails to fetch/deserialize task result

[ https://issues.apache.org/jira/browse/SPARK-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2267. Resolution: Fixed Fix Version/s: 1.0.1 Log exception when TaskResultGetter fails to

[jira] [Resolved] (SPARK-2038) Don't shadow conf variable in saveAsHadoop functions

[ https://issues.apache.org/jira/browse/SPARK-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2038. Resolution: Fixed Don't shadow conf variable in saveAsHadoop functions

[jira] [Created] (SPARK-2275) More general Storage Interface for Shuffle / Spill etc.

Raymond Liu created SPARK-2275: -- Summary: More general Storage Interface for Shuffle / Spill etc. Key: SPARK-2275 URL: https://issues.apache.org/jira/browse/SPARK-2275 Project: Spark Issue

[jira] [Commented] (SPARK-2251) MLLib Naive Bayes Example SparkException: Can only zip RDDs with same number of elements in each partition

[ https://issues.apache.org/jira/browse/SPARK-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043169#comment-14043169 ] Sean Owen commented on SPARK-2251: -- Well the change to the examples is pretty

[jira] [Commented] (SPARK-1127) Add saveAsHBase to PairRDDFunctions

[ https://issues.apache.org/jira/browse/SPARK-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043170#comment-14043170 ] Reynold Xin commented on SPARK-1127: I looked at this a little bit. It seems to me it

[jira] [Commented] (SPARK-1247) Support some of the RDD double functions on float and int as well.

[ https://issues.apache.org/jira/browse/SPARK-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043180#comment-14043180 ] Ximo Guanter commented on SPARK-1247: - This seems to be fixed by the following

[jira] [Commented] (SPARK-1127) Add saveAsHBase to PairRDDFunctions

[ https://issues.apache.org/jira/browse/SPARK-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043199#comment-14043199 ] haosdent commented on SPARK-1127: - [~rxin] I have added SchemaRDD support. Could you

[jira] [Created] (SPARK-2276) user should be able to provide schema for table creation

Kuldeep created SPARK-2276: -- Summary: user should be able to provide schema for table creation Key: SPARK-2276 URL: https://issues.apache.org/jira/browse/SPARK-2276 Project: Spark Issue Type:

[jira] [Updated] (SPARK-1983) Expose private `inferSchema` method in SQLContext for Scala and Java API

[ https://issues.apache.org/jira/browse/SPARK-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuldeep updated SPARK-1983: --- Labels: easyfix (was: ) Expose private `inferSchema` method in SQLContext for Scala and Java API

[jira] [Commented] (SPARK-2186) Spark SQL DSL support for simple aggregations such as SUM and AVG

[ https://issues.apache.org/jira/browse/SPARK-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043438#comment-14043438 ] Ximo Guanter commented on SPARK-2186: - https://github.com/apache/spark/pull/1211

[jira] [Created] (SPARK-2277) Make TaskScheduler track whether there's host on a rack

Rui Li created SPARK-2277: - Summary: Make TaskScheduler track whether there's host on a rack Key: SPARK-2277 URL: https://issues.apache.org/jira/browse/SPARK-2277 Project: Spark Issue Type:

[jira] [Created] (SPARK-2278) groupBy should support custom comparator

Hans Uhlig created SPARK-2278: - Summary: groupBy should support custom comparator Key: SPARK-2278 URL: https://issues.apache.org/jira/browse/SPARK-2278 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-2278) groupBy should support custom comparator

[ https://issues.apache.org/jira/browse/SPARK-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043694#comment-14043694 ] Hans Uhlig commented on SPARK-2278: --- See

[jira] [Resolved] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-2268. --- Resolution: Invalid Looking at the logs from the user there are multiple exceptions being

[jira] [Created] (SPARK-2279) SparkContext should allow creation of EmptyRDD

Hans Uhlig created SPARK-2279: - Summary: SparkContext should allow creation of EmptyRDD Key: SPARK-2279 URL: https://issues.apache.org/jira/browse/SPARK-2279 Project: Spark Issue Type: New

[jira] [Created] (SPARK-2280) Java Scala reference docs should describe function reference behavior.

Hans Uhlig created SPARK-2280: - Summary: Java Scala reference docs should describe function reference behavior. Key: SPARK-2280 URL: https://issues.apache.org/jira/browse/SPARK-2280 Project: Spark

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize 10, task results bigger than 10MiB block execution

[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043730#comment-14043730 ] Bharath Ravi Kumar commented on SPARK-1112: --- Can a clear workaround be specified

[jira] [Comment Edited] (SPARK-1112) When spark.akka.frameSize 10, task results bigger than 10MiB block execution

[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043730#comment-14043730 ] Bharath Ravi Kumar edited comment on SPARK-1112 at 6/25/14 4:51 PM:

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize 10, task results bigger than 10MiB block execution

[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043738#comment-14043738 ] Patrick Wendell commented on SPARK-1112: [~reachbach] If you are running on

[jira] [Reopened] (SPARK-1112) When spark.akka.frameSize 10, task results bigger than 10MiB block execution

[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-1112: This is not resolved yet because it needs to be back ported into 0.9 When

[jira] [Updated] (SPARK-2269) Clean up and add unit tests for resourceOffers in MesosSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2269: --- Description: This function could be simplified a bit. We could re-write it without

[jira] [Updated] (SPARK-2279) JavaSparkContext should allow creation of EmptyRDD

[ https://issues.apache.org/jira/browse/SPARK-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hans Uhlig updated SPARK-2279: -- Summary: JavaSparkContext should allow creation of EmptyRDD (was: SparkContext should allow creation

[jira] [Resolved] (SPARK-2242) Running sc.parallelize(..).count() hangs pyspark

[ https://issues.apache.org/jira/browse/SPARK-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2242. -- Resolution: Fixed Fix Version/s: 1.1.0 Fixed in

[jira] [Comment Edited] (SPARK-2192) Examples Data Not in Binary Distribution

[ https://issues.apache.org/jira/browse/SPARK-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042938#comment-14042938 ] Henry Saputra edited comment on SPARK-2192 at 6/25/14 5:51 PM:

[jira] [Commented] (SPARK-2192) Examples Data Not in Binary Distribution

[ https://issues.apache.org/jira/browse/SPARK-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043847#comment-14043847 ] Pat McDonough commented on SPARK-2192: -- Based on a very quick and not thorough

[jira] [Commented] (SPARK-1000) Crash when running SparkPi example with local-cluster

[ https://issues.apache.org/jira/browse/SPARK-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043908#comment-14043908 ] Henry Saputra commented on SPARK-1000: -- I am not able to repro this with master

[jira] [Resolved] (SPARK-2270) Kryo cannot serialize results returned by asJavaIterable (and thus groupBy/cogroup are broken in Java APIs when Kryo is used)

[ https://issues.apache.org/jira/browse/SPARK-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2270. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 Issue resolved by

[jira] [Resolved] (SPARK-2204) Scheduler for Mesos in fine-grained mode launches tasks on wrong executors

[ https://issues.apache.org/jira/browse/SPARK-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2204. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 Issue resolved by

[jira] [Updated] (SPARK-1912) Compression memory issue during reduce

[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1912: --- Fix Version/s: 0.9.2 Compression memory issue during reduce

[jira] [Updated] (SPARK-1912) Compression memory issue during reduce

[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1912: --- Fix Version/s: 1.0.1 Compression memory issue during reduce

[jira] [Commented] (SPARK-534) Make SparkContext thread-safe

[ https://issues.apache.org/jira/browse/SPARK-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044023#comment-14044023 ] Hans Uhlig commented on SPARK-534: -- Specific usecase. Executing jobs from Tomcat or other

[jira] [Comment Edited] (SPARK-534) Make SparkContext thread-safe

[ https://issues.apache.org/jira/browse/SPARK-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044023#comment-14044023 ] Hans Uhlig edited comment on SPARK-534 at 6/25/14 8:49 PM: ---

[jira] [Commented] (SPARK-2279) JavaSparkContext should allow creation of EmptyRDD

[ https://issues.apache.org/jira/browse/SPARK-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044063#comment-14044063 ] Patrick Wendell commented on SPARK-2279: I think `EmtpyRDD` is a mostly internal

[jira] [Commented] (SPARK-2266) Log page on Worker UI displays Some(app-id)

[ https://issues.apache.org/jira/browse/SPARK-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044070#comment-14044070 ] Patrick Wendell commented on SPARK-2266: Resolved by:

[jira] [Resolved] (SPARK-2266) Log page on Worker UI displays Some(app-id)

[ https://issues.apache.org/jira/browse/SPARK-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2266. Resolution: Fixed Fix Version/s: 1.0.1 Log page on Worker UI displays Some(app-id)

[jira] [Commented] (SPARK-985) Support Job Cancellation on Mesos Scheduler

[ https://issues.apache.org/jira/browse/SPARK-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044079#comment-14044079 ] Patrick Wendell commented on SPARK-985: --- Some more notes on this from a related

[jira] [Updated] (SPARK-985) Support Job Cancellation on Mesos Scheduler

[ https://issues.apache.org/jira/browse/SPARK-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-985: -- Component/s: Mesos Support Job Cancellation on Mesos Scheduler

[jira] [Commented] (SPARK-2233) make-distribution script should list the git hash in the RELEASE file

[ https://issues.apache.org/jira/browse/SPARK-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044128#comment-14044128 ] Guillaume Ballet commented on SPARK-2233: - Submitted pull request:

[jira] [Created] (SPARK-2282) PySpark crashes if too many tasks complete quickly

Aaron Davidson created SPARK-2282: - Summary: PySpark crashes if too many tasks complete quickly Key: SPARK-2282 URL: https://issues.apache.org/jira/browse/SPARK-2282 Project: Spark Issue

[jira] [Updated] (SPARK-2204) Scheduler for Mesos in fine-grained mode launches tasks on wrong executors

[ https://issues.apache.org/jira/browse/SPARK-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2204: --- Assignee: Sebastien Rainville Scheduler for Mesos in fine-grained mode launches tasks on

[jira] [Created] (SPARK-2283) PruningSuite fails if run right after HiveCompatibilitySuite

Cheng Lian created SPARK-2283: - Summary: PruningSuite fails if run right after HiveCompatibilitySuite Key: SPARK-2283 URL: https://issues.apache.org/jira/browse/SPARK-2283 Project: Spark Issue

[jira] [Updated] (SPARK-2258) Worker UI displays zombie executors

[ https://issues.apache.org/jira/browse/SPARK-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2258: - Fix Version/s: 1.1.0 Worker UI displays zombie executors ---

[jira] [Commented] (SPARK-2172) PySpark cannot import mllib modules in YARN-client mode

[ https://issues.apache.org/jira/browse/SPARK-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044197#comment-14044197 ] Piotr Szul commented on SPARK-2172: --- I got the same problem while runing the kmeans.py

[jira] [Commented] (SPARK-2172) PySpark cannot import mllib modules in YARN-client mode

[ https://issues.apache.org/jira/browse/SPARK-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044200#comment-14044200 ] Piotr Szul commented on SPARK-2172: --- I believe the problem is that the

[jira] [Comment Edited] (SPARK-2172) PySpark cannot import mllib modules in YARN-client mode

[ https://issues.apache.org/jira/browse/SPARK-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044200#comment-14044200 ] Piotr Szul edited comment on SPARK-2172 at 6/26/14 12:02 AM: -

[jira] [Resolved] (SPARK-2258) Worker UI displays zombie executors

[ https://issues.apache.org/jira/browse/SPARK-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-2258. -- Resolution: Fixed https://github.com/apache/spark/pull/1213 Worker UI displays zombie executors

[jira] [Created] (SPARK-2284) Failed tasks reported as success if the failure reason is not ExceptionFailure

Reynold Xin created SPARK-2284: -- Summary: Failed tasks reported as success if the failure reason is not ExceptionFailure Key: SPARK-2284 URL: https://issues.apache.org/jira/browse/SPARK-2284 Project:

[jira] [Commented] (SPARK-2213) Sort Merge Join

[ https://issues.apache.org/jira/browse/SPARK-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044279#comment-14044279 ] Cheng Hao commented on SPARK-2213: -- This probably depends on

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044285#comment-14044285 ] Saisai Shao commented on SPARK-2125: PR submitted:

[jira] [Created] (SPARK-2285) Give various TaskEndReason subclass more descriptive names

Reynold Xin created SPARK-2285: -- Summary: Give various TaskEndReason subclass more descriptive names Key: SPARK-2285 URL: https://issues.apache.org/jira/browse/SPARK-2285 Project: Spark Issue

[jira] [Resolved] (SPARK-1749) DAGScheduler supervisor strategy broken with Mesos

[ https://issues.apache.org/jira/browse/SPARK-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1749. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 Issue resolved by

[jira] [Updated] (SPARK-2172) PySpark cannot import mllib modules in YARN-client mode

[ https://issues.apache.org/jira/browse/SPARK-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2172: - Fix Version/s: 1.1.0 1.0.1 PySpark cannot import mllib modules in

[jira] [Resolved] (SPARK-2172) PySpark cannot import mllib modules in YARN-client mode

[ https://issues.apache.org/jira/browse/SPARK-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2172. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/1223 by [~piotrszul] .

[jira] [Created] (SPARK-2286) Report exception/errors for failed tasks that are not ExceptionFailure

Reynold Xin created SPARK-2286: -- Summary: Report exception/errors for failed tasks that are not ExceptionFailure Key: SPARK-2286 URL: https://issues.apache.org/jira/browse/SPARK-2286 Project: Spark

[jira] [Updated] (SPARK-2284) Failed tasks reported as success if the failure reason is not ExceptionFailure

[ https://issues.apache.org/jira/browse/SPARK-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2284: --- Fix Version/s: 1.1.0 1.0.1 Failed tasks reported as success if the failure

[jira] [Resolved] (SPARK-2284) Failed tasks reported as success if the failure reason is not ExceptionFailure

[ https://issues.apache.org/jira/browse/SPARK-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2284. Resolution: Fixed Target Version/s: 1.0.1, 1.1.0 (was: 1.1.0) Failed tasks reported as