[jira] [Created] (SPARK-15915) CacheManager should use canonicalized plan for planToCache.

2016-06-13 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15915: - Summary: CacheManager should use canonicalized plan for planToCache. Key: SPARK-15915 URL: https://issues.apache.org/jira/browse/SPARK-15915 Project: Spark

[jira] [Commented] (SPARK-15915) CacheManager should use canonicalized plan for planToCache.

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326921#comment-15326921 ] Apache Spark commented on SPARK-15915: -- User 'ueshin' has created a pull request for

[jira] [Assigned] (SPARK-15915) CacheManager should use canonicalized plan for planToCache.

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15915: Assignee: (was: Apache Spark) > CacheManager should use canonicalized plan for planToC

[jira] [Assigned] (SPARK-15915) CacheManager should use canonicalized plan for planToCache.

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15915: Assignee: Apache Spark > CacheManager should use canonicalized plan for planToCache. > ---

[jira] [Created] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Piotr Czarnas (JIRA)
Piotr Czarnas created SPARK-15916: - Summary: JDBC AND/OR operator push down does not respect lower OR operator precedence Key: SPARK-15916 URL: https://issues.apache.org/jira/browse/SPARK-15916 Projec

[jira] [Commented] (SPARK-14503) spark.ml API for FPGrowth

2016-06-13 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326977#comment-15326977 ] Jeff Zhang commented on SPARK-14503: [~GayathriMurali] [~yuhaoyan] Do you still work

[jira] [Commented] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326979#comment-15326979 ] Sean Owen commented on SPARK-15796: --- A new parameter like that would just be going back

[jira] [Commented] (SPARK-14503) spark.ml API for FPGrowth

2016-06-13 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326994#comment-15326994 ] yuhao yang commented on SPARK-14503: Hi Jeff, welcome to contribute. I'm discussing

[jira] [Updated] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15796: -- Priority: Blocker (was: Minor) Pardon marking this "Blocker", but I think this needs some attention be

[jira] [Resolved] (SPARK-15813) Spark Dyn Allocation Cancel log message misleading

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15813. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13552 [https://github.co

[jira] [Updated] (SPARK-15813) Spark Dyn Allocation Cancel log message misleading

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15813: -- Assignee: Peter Ableda > Spark Dyn Allocation Cancel log message misleading > -

[jira] [Updated] (SPARK-15813) Spark Dyn Allocation Cancel log message misleading

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15813: -- Issue Type: Improvement (was: Bug) > Spark Dyn Allocation Cancel log message misleading >

[jira] [Updated] (SPARK-6320) Adding new query plan strategy to SQLContext

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6320: - Assignee: Takuya Ueshin > Adding new query plan strategy to SQLContext > -

[jira] [Updated] (SPARK-15788) PySpark IDFModel missing "idf" property

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15788: -- Assignee: Jeff Zhang > PySpark IDFModel missing "idf" property > --

[jira] [Updated] (SPARK-15489) Dataset kryo encoder won't load custom user settings

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15489: -- Assignee: Amit Sela > Dataset kryo encoder won't load custom user settings > -

[jira] [Updated] (SPARK-15743) Prevent saving with all-column partitioning

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15743: -- Assignee: Dongjoon Hyun > Prevent saving with all-column partitioning > ---

[jira] [Commented] (SPARK-15790) Audit @Since annotations in ML

2016-06-13 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327028#comment-15327028 ] Nick Pentreath commented on SPARK-15790: Ah thanks - missed that umbrella. It's a

[jira] [Commented] (SPARK-6628) ClassCastException occurs when executing sql statement "insert into" on hbase table

2016-06-13 Thread Murshid Chalaev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327065#comment-15327065 ] Murshid Chalaev commented on SPARK-6628: Spark 1.6.1 is affected as well, is there

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327082#comment-15327082 ] yuhao yang commented on SPARK-15904: Hi [~Purple]] What's your k and vector size? Btw

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327090#comment-15327090 ] Alessio commented on SPARK-15904: - Hi [~yuhaoyan]], the dataset size is 9120 rows and 212

[jira] [Commented] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327106#comment-15327106 ] Hyukjin Kwon commented on SPARK-15916: -- Indeed. Do you mind if I submit a PR for thi

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327108#comment-15327108 ] yuhao yang commented on SPARK-15904: Thanks for reporting it. I'm not sure if the iss

[jira] [Created] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property

2016-06-13 Thread Jonathan Taws (JIRA)
Jonathan Taws created SPARK-15917: - Summary: Define the number of executors in standalone mode with an easy-to-use property Key: SPARK-15917 URL: https://issues.apache.org/jira/browse/SPARK-15917 Proj

[jira] [Updated] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property

2016-06-13 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Taws updated SPARK-15917: -- Description: After stumbling across a few StackOverflow posts around the issue of using a fixe

[jira] [Updated] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio updated SPARK-15904: Issue Type: Improvement (was: Bug) > High Memory Pressure using MLlib K-means > --

[jira] [Updated] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio updated SPARK-15904: Priority: Minor (was: Major) > High Memory Pressure using MLlib K-means >

[jira] [Commented] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Piotr Czarnas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327118#comment-15327118 ] Piotr Czarnas commented on SPARK-15916: --- Hi, I wish so. This issue is failing a lo

[jira] [Commented] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327144#comment-15327144 ] Apache Spark commented on SPARK-15916: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15916: Assignee: (was: Apache Spark) > JDBC AND/OR operator push down does not respect lower

[jira] [Assigned] (SPARK-15916) JDBC AND/OR operator push down does not respect lower OR operator precedence

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15916: Assignee: Apache Spark > JDBC AND/OR operator push down does not respect lower OR operator

[jira] [Reopened] (SPARK-15345) SparkSession's conf doesn't take effect when there's already an existing SparkContext

2016-06-13 Thread Piotr Milanowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Milanowski reopened SPARK-15345: -- Does not work as expected when using spark-submit; for example, this works fine and prints

[jira] [Created] (SPARK-15918) unionAll returns wrong result when two dataframes has schema in different order

2016-06-13 Thread Prabhu Joseph (JIRA)
Prabhu Joseph created SPARK-15918: - Summary: unionAll returns wrong result when two dataframes has schema in different order Key: SPARK-15918 URL: https://issues.apache.org/jira/browse/SPARK-15918 Pro

[jira] [Created] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Aamir Abbas (JIRA)
Aamir Abbas created SPARK-15919: --- Summary: DStream "saveAsTextFile" doesn't update the prefix after each checkpoint Key: SPARK-15919 URL: https://issues.apache.org/jira/browse/SPARK-15919 Project: Spark

[jira] [Commented] (SPARK-8546) PMML export for Naive Bayes

2016-06-13 Thread Radoslaw Gasiorek (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327167#comment-15327167 ] Radoslaw Gasiorek commented on SPARK-8546: -- hi there, [~josephkb] We would like t

[jira] [Created] (SPARK-15920) Using map on DataFrame

2016-06-13 Thread Piotr Milanowski (JIRA)
Piotr Milanowski created SPARK-15920: Summary: Using map on DataFrame Key: SPARK-15920 URL: https://issues.apache.org/jira/browse/SPARK-15920 Project: Spark Issue Type: Bug Comp

[jira] [Closed] (SPARK-15293) 'collect_list' function undefined

2016-06-13 Thread Piotr Milanowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Milanowski closed SPARK-15293. Works fine, thanks. > 'collect_list' function undefined > - >

[jira] [Created] (SPARK-15921) Spark unable to read partitioned table in avro format and column name in upper case

2016-06-13 Thread Rajkumar Singh (JIRA)
Rajkumar Singh created SPARK-15921: -- Summary: Spark unable to read partitioned table in avro format and column name in upper case Key: SPARK-15921 URL: https://issues.apache.org/jira/browse/SPARK-15921

[jira] [Updated] (SPARK-15921) Spark unable to read partitioned table in avro format and column name in upper case

2016-06-13 Thread Rajkumar Singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated SPARK-15921: --- Description: Spark return null value if the field name is uppercase in hive avro partitioned

[jira] [Commented] (SPARK-15790) Audit @Since annotations in ML

2016-06-13 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327193#comment-15327193 ] Nick Pentreath commented on SPARK-15790: Yes, I've just looked at things in the c

[jira] [Commented] (SPARK-10258) Add @Since annotation to ml.feature

2016-06-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327197#comment-15327197 ] Apache Spark commented on SPARK-10258: -- User 'MLnick' has created a pull request for

[jira] [Commented] (SPARK-6628) ClassCastException occurs when executing sql statement "insert into" on hbase table

2016-06-13 Thread Teng Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327201#comment-15327201 ] Teng Qiu commented on SPARK-6628: - this is caused by missing interface implementation in

[jira] [Resolved] (SPARK-15920) Using map on DataFrame

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15920. --- Resolution: Not A Problem Target Version/s: (was: 2.0.0) Don't set Target please, and thi

[jira] [Commented] (SPARK-8546) PMML export for Naive Bayes

2016-06-13 Thread Villu Ruusmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327205#comment-15327205 ] Villu Ruusmann commented on SPARK-8546: --- Hi [~rgasiorek] - would it be an option to

[jira] [Commented] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread binde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327209#comment-15327209 ] binde commented on SPARK-15919: --- this is not a bug, getOutputPath() will be invoked on the

[jira] [Commented] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Aamir Abbas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327212#comment-15327212 ] Aamir Abbas commented on SPARK-15919: - I need to save the output of each batch in a d

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327220#comment-15327220 ] Nick Pentreath commented on SPARK-15904: Could you explain why you're using K>300

[jira] [Commented] (SPARK-6628) ClassCastException occurs when executing sql statement "insert into" on hbase table

2016-06-13 Thread Murshid Chalaev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327224#comment-15327224 ] Murshid Chalaev commented on SPARK-6628: Thank you > ClassCastException occurs wh

[jira] [Resolved] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15919. --- Resolution: Not A Problem No, this is simple to accomplish in Spark already. You need to use foreachR

[jira] [Commented] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Aamir Abbas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327229#comment-15327229 ] Aamir Abbas commented on SPARK-15919: - ForeachRDD is fine in case you want to save in

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327234#comment-15327234 ] Alessio commented on SPARK-15904: - My dataset has 9000+ patterns, each of which has 2000+

[jira] [Commented] (SPARK-12623) map key_values to values

2016-06-13 Thread Elazar Gershuni (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327236#comment-15327236 ] Elazar Gershuni commented on SPARK-12623: - At the very least, it should have a "w

[jira] [Commented] (SPARK-15746) SchemaUtils.checkColumnType with VectorUDT prints instance details in error message

2016-06-13 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327237#comment-15327237 ] Nick Pentreath commented on SPARK-15746: I think you can go ahead now - I also vo

[jira] [Reopened] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Aamir Abbas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aamir Abbas reopened SPARK-15919: - This is an issue, as I do not actually need the current timestamp to use in output path. I need the

[jira] [Commented] (SPARK-12623) map key_values to values

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327248#comment-15327248 ] Sean Owen commented on SPARK-12623: --- The Status can only be "Resolved". You're referrin

[jira] [Closed] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-15919. - > DStream "saveAsTextFile" doesn't update the prefix after each checkpoint >

[jira] [Resolved] (SPARK-15919) DStream "saveAsTextFile" doesn't update the prefix after each checkpoint

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15919. --- Resolution: Not A Problem Look at the implementation of DStream.saveAsTextFiles -- about all it does

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327255#comment-15327255 ] Sean Owen commented on SPARK-15904: --- Yeah it's coherent, though typically k << number o

[jira] [Updated] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessio updated SPARK-15904: Description: Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on Memory and Disk. Ever

[jira] [Closed] (SPARK-15921) Spark unable to read partitioned table in avro format and column name in upper case

2016-06-13 Thread Rajkumar Singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh closed SPARK-15921. -- Resolution: Fixed > Spark unable to read partitioned table in avro format and column name in >

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327272#comment-15327272 ] Alessio commented on SPARK-15904: - Dear Sean, I must certainly agree with you on k< High

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327272#comment-15327272 ] Alessio edited comment on SPARK-15904 at 6/13/16 12:41 PM: --- Dea

[jira] [Comment Edited] (SPARK-8546) PMML export for Naive Bayes

2016-06-13 Thread Radoslaw Gasiorek (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327167#comment-15327167 ] Radoslaw Gasiorek edited comment on SPARK-8546 at 6/13/16 12:43 PM:

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327272#comment-15327272 ] Alessio edited comment on SPARK-15904 at 6/13/16 12:44 PM: --- Dea

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327272#comment-15327272 ] Alessio edited comment on SPARK-15904 at 6/13/16 12:45 PM: --- Dea

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327369#comment-15327369 ] Sean Owen commented on SPARK-15904: --- -verbose:gc is a JVM option and should write to st

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327397#comment-15327397 ] Alessio commented on SPARK-15904: - Dear [~srowen], at the beginning I noticed that "Clea

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327397#comment-15327397 ] Alessio edited comment on SPARK-15904 at 6/13/16 1:49 PM: -- Dear

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327405#comment-15327405 ] Sean Owen commented on SPARK-15904: --- Hm, but that only means Spark used a lot of memory

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327397#comment-15327397 ] Alessio edited comment on SPARK-15904 at 6/13/16 1:48 PM: -- Dear

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327411#comment-15327411 ] Alessio commented on SPARK-15904: - This is absolutely weird to me. I gave Spark 9GB and d

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327430#comment-15327430 ] Sean Owen commented on SPARK-15904: --- How much RAM does your machine have? 10GB heap mea

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327411#comment-15327411 ] Alessio edited comment on SPARK-15904 at 6/13/16 1:55 PM: -- This

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327438#comment-15327438 ] Alessio commented on SPARK-15904: - My machine has 16GB of RAM. I also tried closing all t

[jira] [Resolved] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15904. --- Resolution: Not A Problem Memory and disk still means it's also persisting in memory. I think you'll

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327443#comment-15327443 ] Alessio commented on SPARK-15904: - Correct. Memory and Disk gives priority to Memory...bu

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327449#comment-15327449 ] Sean Owen commented on SPARK-15904: --- It's not your 400MB data set that is the only thin

[jira] [Created] (SPARK-15922) BlockMatrix to IndexedRowMatrix throws an error

2016-06-13 Thread Charlie Evans (JIRA)
Charlie Evans created SPARK-15922: - Summary: BlockMatrix to IndexedRowMatrix throws an error Key: SPARK-15922 URL: https://issues.apache.org/jira/browse/SPARK-15922 Project: Spark Issue Type:

[jira] [Updated] (SPARK-15922) BlockMatrix to IndexedRowMatrix throws an error

2016-06-13 Thread Charlie Evans (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Evans updated SPARK-15922: -- Description: {code} import org.apache.spark.mllib.linalg.distributed._ import org.apache.spark.

[jira] [Updated] (SPARK-15922) BlockMatrix to IndexedRowMatrix throws an error

2016-06-13 Thread Charlie Evans (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Evans updated SPARK-15922: -- Description: {code} import org.apache.spark.mllib.linalg.distributed._ import org.apache.spark.

[jira] [Updated] (SPARK-15918) unionAll returns wrong result when two dataframes has schema in different order

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15918: -- Fix Version/s: (was: 1.6.1) Don't set fix version; 1.6.1 wouldn't make sense anyway. > unionAll re

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327476#comment-15327476 ] Alessio commented on SPARK-15904: - If anyone's interested, the dataset I'm working on is

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327510#comment-15327510 ] Sean Owen commented on SPARK-15904: --- Yes, that just means "out of memory". The question

[jira] [Comment Edited] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327476#comment-15327476 ] Alessio edited comment on SPARK-15904 at 6/13/16 2:48 PM: -- If an

[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means

2016-06-13 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327542#comment-15327542 ] Alessio commented on SPARK-15904: - With the --driver-memory 4G switch I've tried both. Wi

[jira] [Commented] (SPARK-15118) spark couldn't get hive properyties in hive-site.xml

2016-06-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327674#comment-15327674 ] Herman van Hovell commented on SPARK-15118: --- [~eksmile] any update on this? >

[jira] [Commented] (SPARK-15370) Some correlated subqueries return incorrect answers

2016-06-13 Thread Luciano Resende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327683#comment-15327683 ] Luciano Resende commented on SPARK-15370: - [~hvanhovell] You might need to add [~

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

2016-06-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327703#comment-15327703 ] Herman van Hovell commented on SPARK-15822: --- [~robbinspg] You can dump the plan

[jira] [Commented] (SPARK-15902) Add a deprecation warning for Python 2.6

2016-06-13 Thread Krishna Kalyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327723#comment-15327723 ] Krishna Kalyan commented on SPARK-15902: Hi [~holdenk], I have some questions, wh

[jira] [Created] (SPARK-15923) Spark Application rest api returns "no such app: "

2016-06-13 Thread Yesha Vora (JIRA)
Yesha Vora created SPARK-15923: -- Summary: Spark Application rest api returns "no such app: " Key: SPARK-15923 URL: https://issues.apache.org/jira/browse/SPARK-15923 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

2016-06-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327750#comment-15327750 ] Sean Owen commented on SPARK-15923: --- [~tgraves] or [~ste...@apache.org] will probably k

[jira] [Resolved] (SPARK-15814) Aggregator can return null result

2016-06-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-15814. --- Resolution: Resolved > Aggregator can return null result > --

[jira] [Commented] (SPARK-15163) Mark experimental algorithms experimental in PySpark

2016-06-13 Thread Krishna Kalyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327748#comment-15327748 ] Krishna Kalyan commented on SPARK-15163: Hi [~holdenk], Is this task still up for

[jira] [Created] (SPARK-15924) SparkR parser bug with backslash in comments

2016-06-13 Thread Xuan Wang (JIRA)
Xuan Wang created SPARK-15924: - Summary: SparkR parser bug with backslash in comments Key: SPARK-15924 URL: https://issues.apache.org/jira/browse/SPARK-15924 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15924) SparkR parser bug with backslash in comments

2016-06-13 Thread Xuan Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Wang updated SPARK-15924: -- Description: When I run an R cell with the following comments: {code} # p <- p + scale_fill_manual(v

[jira] [Updated] (SPARK-15924) SparkR parser bug with backslash in comments

2016-06-13 Thread Xuan Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Wang updated SPARK-15924: -- Description: When I run an R cell with the following comments: {code} # p <- p + scale_fill_manual(v

[jira] [Updated] (SPARK-15924) SparkR parser bug with backslash in comments

2016-06-13 Thread Xuan Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Wang updated SPARK-15924: -- Description: When I run an R cell with the following comments: {code} # p <- p + scale_fill_manual(v

[jira] [Resolved] (SPARK-15913) Dispatcher.stopped should be enclosed by synchronized block.

2016-06-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15913. Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.0.0 > Dispatcher

[jira] [Updated] (SPARK-15826) PipedRDD to allow configurable char encoding (default: UTF-8)

2016-06-13 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tejas Patil updated SPARK-15826: Summary: PipedRDD to allow configurable char encoding (default: UTF-8) (was: PipedRDD to strictly

[jira] [Commented] (SPARK-15345) SparkSession's conf doesn't take effect when there's already an existing SparkContext

2016-06-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327803#comment-15327803 ] Herman van Hovell commented on SPARK-15345: --- [~m1lan] Just to be sure, is this

[jira] [Commented] (SPARK-15666) Join on two tables generated from a same table throwing query analyzer issue

2016-06-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327818#comment-15327818 ] Herman van Hovell commented on SPARK-15666: --- [~mkbond777] Is this also a proble

  1   2   3   >