[jira] [Created] (SPARK-5426) SchemaRDD is java incompatible

2015-01-27 Thread Kuldeep (JIRA)
Kuldeep created SPARK-5426: -- Summary: SchemaRDD is java incompatible Key: SPARK-5426 URL: https://issues.apache.org/jira/browse/SPARK-5426 Project: Spark Issue Type: Bug Components: SQL

[jira] [Commented] (SPARK-5051) python: module pyspark.daemon not found

2015-01-27 Thread naveen kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293190#comment-14293190 ] naveen kumar commented on SPARK-5051: - Yes i have installed python on that location

[jira] [Commented] (SPARK-3439) Add Canopy Clustering Algorithm

2015-01-27 Thread Muhammad-Ali A'rabi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293526#comment-14293526 ] Muhammad-Ali A'rabi commented on SPARK-3439: Input is a collection of vectors

[jira] [Created] (SPARK-5427) Add support for floor function in Spark SQL

2015-01-27 Thread Ted Yu (JIRA)
Ted Yu created SPARK-5427: - Summary: Add support for floor function in Spark SQL Key: SPARK-5427 URL: https://issues.apache.org/jira/browse/SPARK-5427 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5430) Move treeReduce and treeAggregate to core

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294000#comment-14294000 ] Apache Spark commented on SPARK-5430: - User 'mengxr' has created a pull request for

[jira] [Created] (SPARK-5433) Spark EC2 doesn't mount local disks for some instance types

2015-01-27 Thread Tomer Kaftan (JIRA)
Tomer Kaftan created SPARK-5433: --- Summary: Spark EC2 doesn't mount local disks for some instance types Key: SPARK-5433 URL: https://issues.apache.org/jira/browse/SPARK-5433 Project: Spark

[jira] [Resolved] (SPARK-5308) MD5 / SHA1 hash format doesn't match standard Maven output

2015-01-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5308. Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Assignee:

[jira] [Resolved] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-01-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5299. Resolution: Fixed Thanks I've fixed this. Is http://www.apache.org/dist/spark/KEYS out of

[jira] [Commented] (SPARK-5155) Python API for MQTT streaming

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294006#comment-14294006 ] Apache Spark commented on SPARK-5155: - User 'prabeesh' has created a pull request for

[jira] [Updated] (SPARK-5431) SparkSubmitSuite and DriverSuite hang indefinitely if Master fails

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5431: - Priority: Major (was: Critical) SparkSubmitSuite and DriverSuite hang indefinitely if Master fails

[jira] [Updated] (SPARK-5431) SparkSubmitSuite and DriverSuite hang indefinitely if Master fails

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5431: - Affects Version/s: 1.0.0 SparkSubmitSuite and DriverSuite hang indefinitely if Master fails

[jira] [Commented] (SPARK-5432) DriverSuite and SparkSubmitSuite should sc.stop()

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294080#comment-14294080 ] Apache Spark commented on SPARK-5432: - User 'andrewor14' has created a pull request

[jira] [Commented] (SPARK-5051) python: module pyspark.daemon not found

2015-01-27 Thread Sven Krasser (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293916#comment-14293916 ] Sven Krasser commented on SPARK-5051: - I assume this is related to this thread:

[jira] [Created] (SPARK-5431) SparkSubmitSuite and DriverSuite hang indefinitely if Master fails

2015-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5431: Summary: SparkSubmitSuite and DriverSuite hang indefinitely if Master fails Key: SPARK-5431 URL: https://issues.apache.org/jira/browse/SPARK-5431 Project: Spark

[jira] [Reopened] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-01-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-5299: Actually I need to deal with past releases as well, so re-opening. Is

[jira] [Commented] (SPARK-4587) Model export/import

2015-01-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294035#comment-14294035 ] Joseph K. Bradley commented on SPARK-4587: -- Thanks for reading the doc! The

[jira] [Commented] (SPARK-5119) java.lang.ArrayIndexOutOfBoundsException on trying to train decision tree model

2015-01-27 Thread Nicolas Garneau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294046#comment-14294046 ] Nicolas Garneau commented on SPARK-5119: Hey guys, I am wondering what you think

[jira] [Updated] (SPARK-5433) Spark EC2 doesn't mount local disks for all instance types

2015-01-27 Thread Tomer Kaftan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomer Kaftan updated SPARK-5433: Description: Launching a cluster using spark-ec2 will currently mount all local disks for the r3

[jira] [Resolved] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-01-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5299. Resolution: Fixed Okay I've now added every key ever used to publish a Spark release (I

[jira] [Updated] (SPARK-5428) Declare the 'assembly' module at the bottom of the modules element in the parent POM

2015-01-27 Thread Christian Tzolov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Tzolov updated SPARK-5428: Description: For multiple-modules projects, Maven follows those execution order rules:

[jira] [Updated] (SPARK-5428) Declare the 'assembly' module at the bottom of the modules element in the parent POM

2015-01-27 Thread Christian Tzolov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Tzolov updated SPARK-5428: Description: For multiple-modules projects, Maven follows those execution order rules:

[jira] [Commented] (SPARK-5097) Adding data frame APIs to SchemaRDD

2015-01-27 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293905#comment-14293905 ] Reynold Xin commented on SPARK-5097: I've debating that myself for a while. The main

[jira] [Created] (SPARK-5430) Move treeReduce and treeAggregate to core

2015-01-27 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-5430: Summary: Move treeReduce and treeAggregate to core Key: SPARK-5430 URL: https://issues.apache.org/jira/browse/SPARK-5430 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4964) Exactly-once + WAL-free Kafka Support in Spark Streaming

2015-01-27 Thread vincent ye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293899#comment-14293899 ] vincent ye commented on SPARK-4964: --- I have pretty much the same idea as mentioned in

[jira] [Commented] (SPARK-1835) sbt gen-idea includes both mesos and mesos with shaded-protobuf into dependencies

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293929#comment-14293929 ] Sean Owen commented on SPARK-1835: -- FWIW I can confirm this still occurs: {code} $ grep

[jira] [Resolved] (SPARK-5419) Fix the logic in Vectors.sqdist

2015-01-27 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-5419. -- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 4217

[jira] [Created] (SPARK-5424) Make the new ALS implementation take generic ID types

2015-01-27 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-5424: Summary: Make the new ALS implementation take generic ID types Key: SPARK-5424 URL: https://issues.apache.org/jira/browse/SPARK-5424 Project: Spark Issue

[jira] [Commented] (SPARK-5162) Python yarn-cluster mode

2015-01-27 Thread Lianhui Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293271#comment-14293271 ] Lianhui Wang commented on SPARK-5162: - [~vgrigor] thank you for asking some questions.

[jira] [Commented] (SPARK-5300) Spark loads file partitions in inconsistent order on native filesystems

2015-01-27 Thread Ewan Higgs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293201#comment-14293201 ] Ewan Higgs commented on SPARK-5300: --- The PR appears to have been rejected on the grounds

[jira] [Created] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Jacek Lewandowski (JIRA)
Jacek Lewandowski created SPARK-5425: Summary: ConcurrentModificationException during SparkConf creation Key: SPARK-5425 URL: https://issues.apache.org/jira/browse/SPARK-5425 Project: Spark

[jira] [Commented] (SPARK-5423) ExternalAppendOnlyMap won't delete temp spilled file if some exception happens during using it

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293173#comment-14293173 ] Apache Spark commented on SPARK-5423: - User 'zsxwing' has created a pull request for

[jira] [Comment Edited] (SPARK-5377) Dynamically add jar into Spark Driver's classpath.

2015-01-27 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293186#comment-14293186 ] Jongyoul Lee edited comment on SPARK-5377 at 1/27/15 8:44 AM: --

[jira] [Commented] (SPARK-5377) Dynamically add jar into Spark Driver's classpath.

2015-01-27 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293186#comment-14293186 ] Jongyoul Lee commented on SPARK-5377: - It's really needed. When we use HiveContext and

[jira] [Commented] (SPARK-3298) [SQL] registerAsTable / registerTempTable overwrites old tables

2015-01-27 Thread shengli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293205#comment-14293205 ] shengli commented on SPARK-3298: @Michael Armbrust Would you please give some comments

[jira] [Commented] (SPARK-5097) Adding data frame APIs to SchemaRDD

2015-01-27 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294193#comment-14294193 ] Sandy Ryza commented on SPARK-5097: --- Ah, yeah, I hadn't considered that aspect. I

[jira] [Updated] (SPARK-5432) SparkSubmitSuite should sc.stop()

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5432: - Summary: SparkSubmitSuite should sc.stop() (was: DriverSuite and SparkSubmitSuite should sc.stop())

[jira] [Created] (SPARK-5437) DriverSuite and SparkSubmitSuite leak JVMs on timeout

2015-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5437: Summary: DriverSuite and SparkSubmitSuite leak JVMs on timeout Key: SPARK-5437 URL: https://issues.apache.org/jira/browse/SPARK-5437 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3381) DecisionTree: eliminate bins for unordered features

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294161#comment-14294161 ] Apache Spark commented on SPARK-3381: - User 'MechCoder' has created a pull request for

[jira] [Created] (SPARK-5435) saveAsNewAPIHadoopDataset is not setting up the local configuration

2015-01-27 Thread Joe Mudd (JIRA)
Joe Mudd created SPARK-5435: --- Summary: saveAsNewAPIHadoopDataset is not setting up the local configuration Key: SPARK-5435 URL: https://issues.apache.org/jira/browse/SPARK-5435 Project: Spark

[jira] [Commented] (SPARK-5428) Declare the 'assembly' module at the bottom of the modules element in the parent POM

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294246#comment-14294246 ] Apache Spark commented on SPARK-5428: - User 'tzolov' has created a pull request for

[jira] [Commented] (SPARK-5338) Support cluster mode with Mesos

2015-01-27 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294264#comment-14294264 ] Timothy Chen commented on SPARK-5338: - Started a doc to begin the design of this, will

[jira] [Updated] (SPARK-5191) Pyspark: scheduler hangs when importing a standalone pyspark app

2015-01-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-5191: -- Component/s: PySpark Pyspark: scheduler hangs when importing a standalone pyspark app

[jira] [Created] (SPARK-5434) Preserve spaces in path to spark-ec2

2015-01-27 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5434: --- Summary: Preserve spaces in path to spark-ec2 Key: SPARK-5434 URL: https://issues.apache.org/jira/browse/SPARK-5434 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-5436) Validate GradientBoostedTrees during training

2015-01-27 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-5436: Summary: Validate GradientBoostedTrees during training Key: SPARK-5436 URL: https://issues.apache.org/jira/browse/SPARK-5436 Project: Spark Issue

[jira] [Updated] (SPARK-5432) SparkSubmitSuite should sc.stop()

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5432: - Priority: Minor (was: Major) SparkSubmitSuite should sc.stop() -

[jira] [Updated] (SPARK-5432) SparkSubmitSuite should sc.stop()

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5432: - Description: The reason why we need all the hacks around the UI ports for these two suites is because we

[jira] [Commented] (SPARK-5434) Preserve spaces in path to spark-ec2

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294139#comment-14294139 ] Apache Spark commented on SPARK-5434: - User 'nchammas' has created a pull request for

[jira] [Updated] (SPARK-5338) Support cluster mode with Mesos

2015-01-27 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated SPARK-5338: Component/s: Mesos Support cluster mode with Mesos ---

[jira] [Commented] (SPARK-5437) DriverSuite and SparkSubmitSuite leak JVMs on timeout

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294293#comment-14294293 ] Apache Spark commented on SPARK-5437: - User 'andrewor14' has created a pull request

[jira] [Updated] (SPARK-5191) Pyspark: scheduler hangs when importing a standalone pyspark app

2015-01-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-5191: -- Description: In a.py: {code} from pyspark import SparkContext sc = SparkContext(local, test spark)

[jira] [Updated] (SPARK-5437) DriverSuite and SparkSubmitSuite incorrect timeout behavior

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5437: - Summary: DriverSuite and SparkSubmitSuite incorrect timeout behavior (was: DriverSuite and

[jira] [Updated] (SPARK-5191) Pyspark: scheduler hangs when importing a standalone pyspark app

2015-01-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-5191: -- Affects Version/s: 1.2.1 1.3.0 1.0.2

[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI

2015-01-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294311#comment-14294311 ] Imran Rashid commented on SPARK-3454: - There are a few ways we could move forward on

[jira] [Commented] (SPARK-5206) Accumulators are not re-registered during recovering from checkpoint

2015-01-27 Thread vincent ye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294339#comment-14294339 ] vincent ye commented on SPARK-5206: --- I suggest to add life-cycle events into DStream,

[jira] [Commented] (SPARK-5191) Pyspark: scheduler hangs when importing a standalone pyspark app

2015-01-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294351#comment-14294351 ] Josh Rosen commented on SPARK-5191: --- I'm investigating this now and it seems to affect

[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI

2015-01-27 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294348#comment-14294348 ] Ryan Williams commented on SPARK-3454: -- +1 to making the web UI a consumer of a JSON

[jira] [Commented] (SPARK-5439) Expose yarn app id for yarn mode

2015-01-27 Thread bc Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294378#comment-14294378 ] bc Wong commented on SPARK-5439: Parsing the stderr for YARN cluster mode isn't a good

[jira] [Commented] (SPARK-5191) Pyspark: scheduler hangs when importing a standalone pyspark app

2015-01-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294400#comment-14294400 ] Josh Rosen commented on SPARK-5191: --- I think this is actually caused by an interaction

[jira] [Commented] (SPARK-5436) Validate GradientBoostedTrees during training

2015-01-27 Thread Chris T (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294403#comment-14294403 ] Chris T commented on SPARK-5436: I think, then, the only addition needed is to retain the

[jira] [Commented] (SPARK-5162) Python yarn-cluster mode

2015-01-27 Thread Lianhui Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294567#comment-14294567 ] Lianhui Wang commented on SPARK-5162: - [~vgrigor] i think that a quick way to resolve

[jira] [Commented] (SPARK-5440) Add toLocalIterator to pyspark rdd

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294469#comment-14294469 ] Apache Spark commented on SPARK-5440: - User 'mnazario' has created a pull request for

[jira] [Commented] (SPARK-5436) Validate GradientBoostedTrees during training

2015-01-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294523#comment-14294523 ] Joseph K. Bradley commented on SPARK-5436: -- That sound good. I think the main

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

2015-01-27 Thread Sven Krasser (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294570#comment-14294570 ] Sven Krasser commented on SPARK-5395: - Some new findings: I can trigger the problem

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

2015-01-27 Thread Mark Khaitman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293653#comment-14293653 ] Mark Khaitman commented on SPARK-5395: -- Actually I think I know why this happens...

[jira] [Commented] (SPARK-1742) Profiler for Spark

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293617#comment-14293617 ] Sean Owen commented on SPARK-1742: -- Is it realistic to expect that this would be part of

[jira] [Commented] (SPARK-5436) Validate GradientBoostedTrees during training

2015-01-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294360#comment-14294360 ] Joseph K. Bradley commented on SPARK-5436: -- Yes, it would be reasonable to take

[jira] [Commented] (SPARK-3290) No unpersist callls in SVDPlusPlus

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294393#comment-14294393 ] Apache Spark commented on SPARK-3290: - User 'srowen' has created a pull request for

[jira] [Resolved] (SPARK-5199) Input metrics should show up for InputFormats that return CombineFileSplits

2015-01-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5199. Resolution: Fixed Fix Version/s: 1.3.0 Input metrics should show up for

[jira] [Created] (SPARK-5440) Add toLocalIterator to pyspark rdd

2015-01-27 Thread Michael Nazario (JIRA)
Michael Nazario created SPARK-5440: -- Summary: Add toLocalIterator to pyspark rdd Key: SPARK-5440 URL: https://issues.apache.org/jira/browse/SPARK-5440 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5097) Adding data frame APIs to SchemaRDD

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294454#comment-14294454 ] Apache Spark commented on SPARK-5097: - User 'rxin' has created a pull request for this

[jira] [Commented] (SPARK-2802) Improve the Cassandra sample and Add a new sample for Streaming to Cassandra

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294314#comment-14294314 ] Sean Owen commented on SPARK-2802: -- Did this materialize? I imagine at this point the

[jira] [Commented] (SPARK-5439) Expose yarn app id for yarn mode

2015-01-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294362#comment-14294362 ] Marcelo Vanzin commented on SPARK-5439: --- Hi bc, can you provide more context here?

[jira] [Commented] (SPARK-4587) Model export/import

2015-01-27 Thread Peter Prettenhofer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293892#comment-14293892 ] Peter Prettenhofer commented on SPARK-4587: --- I read the design document for

[jira] [Commented] (SPARK-5135) Add support for describe [extended] table to DDL in SQLContext

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293994#comment-14293994 ] Apache Spark commented on SPARK-5135: - User 'OopsOutOfMemory' has created a pull

[jira] [Created] (SPARK-5432) DriverSuite and SparkSubmitSuite should sc.stop()

2015-01-27 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5432: Summary: DriverSuite and SparkSubmitSuite should sc.stop() Key: SPARK-5432 URL: https://issues.apache.org/jira/browse/SPARK-5432 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-5433) Spark EC2 doesn't mount local disks for all instance types

2015-01-27 Thread Tomer Kaftan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomer Kaftan updated SPARK-5433: Summary: Spark EC2 doesn't mount local disks for all instance types (was: Spark EC2 doesn't mount

[jira] [Commented] (SPARK-2559) Add A Link to Download the Application Events Log for Offline Analysis

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294306#comment-14294306 ] Sean Owen commented on SPARK-2559: -- I think this duplicates SPARK-2450 Add A Link to

[jira] [Commented] (SPARK-4587) Model export/import

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294312#comment-14294312 ] Apache Spark commented on SPARK-4587: - User 'jkbradley' has created a pull request for

[jira] [Commented] (SPARK-5438) Graph aggregateMessages should support zero/seqOp/combOp

2015-01-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294319#comment-14294319 ] Joseph K. Bradley commented on SPARK-5438: -- Note: My use case for this is in LDA:

[jira] [Created] (SPARK-5438) Graph aggregateMessages should support zero/seqOp/combOp

2015-01-27 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-5438: Summary: Graph aggregateMessages should support zero/seqOp/combOp Key: SPARK-5438 URL: https://issues.apache.org/jira/browse/SPARK-5438 Project: Spark

[jira] [Resolved] (SPARK-2751) Geeting error to run graphx.LiveJournalPageRank

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2751. -- Resolution: Not a Problem The correct usage is --numEPart=n, which works and is what's documented. You

[jira] [Created] (SPARK-5441) SerDeUtil Pair RDD to python conversion doesn't accept empty RDDs

2015-01-27 Thread Michael Nazario (JIRA)
Michael Nazario created SPARK-5441: -- Summary: SerDeUtil Pair RDD to python conversion doesn't accept empty RDDs Key: SPARK-5441 URL: https://issues.apache.org/jira/browse/SPARK-5441 Project: Spark

[jira] [Closed] (SPARK-5432) SparkSubmitSuite should sc.stop()

2015-01-27 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-5432. Resolution: Won't Fix It may be intended behavior for the test suites to ensure that the driver does

[jira] [Commented] (SPARK-5436) Validate GradientBoostedTrees during training

2015-01-27 Thread Chris T (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294335#comment-14294335 ] Chris T commented on SPARK-5436: The usual way that GBT models are evaluated is by

[jira] [Resolved] (SPARK-5418) Output directory for shuffle should consider left space of each directory set in conf

2015-01-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-5418. -- Resolution: Duplicate Output directory for shuffle should consider left space of each directory set

[jira] [Created] (SPARK-5439) Expose yarn app id for yarn mode

2015-01-27 Thread bc Wong (JIRA)
bc Wong created SPARK-5439: -- Summary: Expose yarn app id for yarn mode Key: SPARK-5439 URL: https://issues.apache.org/jira/browse/SPARK-5439 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5439) Expose yarn app id for yarn mode

2015-01-27 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294406#comment-14294406 ] Xuefu Zhang commented on SPARK-5439: Yeah. This is certainly something desirable in

[jira] [Commented] (SPARK-5441) SerDeUtil Pair RDD to python conversion doesn't accept empty RDDs

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294465#comment-14294465 ] Apache Spark commented on SPARK-5441: - User 'mnazario' has created a pull request for

[jira] [Created] (SPARK-5442) Docs claim users must explicitly depend on a hadoop client, but it is not actually required.

2015-01-27 Thread Juliet Hougland (JIRA)
Juliet Hougland created SPARK-5442: -- Summary: Docs claim users must explicitly depend on a hadoop client, but it is not actually required. Key: SPARK-5442 URL: https://issues.apache.org/jira/browse/SPARK-5442

[jira] [Commented] (SPARK-2812) convert maven to archetype based build

2015-01-27 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293300#comment-14293300 ] Prashant Sharma commented on SPARK-2812: This can be closed since we chose a

[jira] [Commented] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293417#comment-14293417 ] Apache Spark commented on SPARK-5425: - User 'jacek-lewandowski' has created a pull

[jira] [Commented] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293421#comment-14293421 ] Apache Spark commented on SPARK-5425: - User 'jacek-lewandowski' has created a pull

[jira] [Commented] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293423#comment-14293423 ] Apache Spark commented on SPARK-5425: - User 'jacek-lewandowski' has created a pull

[jira] [Commented] (SPARK-5162) Python yarn-cluster mode

2015-01-27 Thread Vladimir Grigor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293397#comment-14293397 ] Vladimir Grigor commented on SPARK-5162: [~lianhuiwang] Thank you for reply :)

[jira] [Resolved] (SPARK-2812) convert maven to archetype based build

2015-01-27 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-2812. Resolution: Invalid convert maven to archetype based build

[jira] [Updated] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Lewandowski updated SPARK-5425: - Description: This fragment of code: {code} if (loadDefaults) { // Load any spark.*

[jira] [Commented] (SPARK-5425) ConcurrentModificationException during SparkConf creation

2015-01-27 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293424#comment-14293424 ] Jacek Lewandowski commented on SPARK-5425: -- [~joshrosen] can you take a look?

[jira] [Updated] (SPARK-5382) Scripts do not use SPARK_CONF_DIR where they should

2015-01-27 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Lewandowski updated SPARK-5382: - Fix Version/s: 1.2.1 1.3.0 Scripts do not use SPARK_CONF_DIR where

[jira] [Updated] (SPARK-5426) SchemaRDD is java incompatible

2015-01-27 Thread Kuldeep (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuldeep updated SPARK-5426: --- Priority: Major (was: Minor) SchemaRDD is java incompatible --

[jira] [Updated] (SPARK-5426) SchemaRDD is java incompatible

2015-01-27 Thread Kuldeep (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuldeep updated SPARK-5426: --- Priority: Minor (was: Major) SchemaRDD is java incompatible --

  1   2   >