[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Assigned] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12673: Assignee: (was: Apache Spark) > Prepending base URI of job description is missing >

[jira] [Assigned] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12673: Assignee: Apache Spark > Prepending base URI of job description is missing >

[jira] [Commented] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085314#comment-15085314 ] Apache Spark commented on SPARK-12673: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-4210) Add Extra-Trees algorithm to MLlib

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4210. -- Resolution: Won't Fix > Add Extra-Trees algorithm to MLlib > -- > >

[jira] [Resolved] (SPARK-1962) Add RDD cache reference counting

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-1962. -- Resolution: Won't Fix I understand the problem, but referencing counting isn't right semantically

[jira] [Created] (SPARK-12674) Spark on Mesos executor exit incorrect

2016-01-06 Thread astralidea (JIRA)
astralidea created SPARK-12674: -- Summary: Spark on Mesos executor exit incorrect Key: SPARK-12674 URL: https://issues.apache.org/jira/browse/SPARK-12674 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10795) FileNotFoundException while deploying pyspark job on cluster

2016-01-06 Thread NISHAN SATHARASINGHE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085395#comment-15085395 ] NISHAN SATHARASINGHE commented on SPARK-10795: -- Having the same issue . could someone help

[jira] [Created] (SPARK-12675) Spark 1.6.0 executor dies because of ClassCastException and causes timeout

2016-01-06 Thread Alexandru Rosianu (JIRA)
Alexandru Rosianu created SPARK-12675: - Summary: Spark 1.6.0 executor dies because of ClassCastException and causes timeout Key: SPARK-12675 URL: https://issues.apache.org/jira/browse/SPARK-12675

[jira] [Issue Comment Deleted] (SPARK-10795) FileNotFoundException while deploying pyspark job on cluster

2016-01-06 Thread NISHAN SATHARASINGHE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NISHAN SATHARASINGHE updated SPARK-10795: - Comment: was deleted (was: Having the same issue . could someone help here ?) >

[jira] [Resolved] (SPARK-12659) NPE when spill in CartisianProduct

2016-01-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-12659. Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 (was: 1.6.1) >

[jira] [Updated] (SPARK-12659) NPE when spill in CartisianProduct

2016-01-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-12659: --- Affects Version/s: (was: 1.6.0) 2.0.0 > NPE when spill in

[jira] [Resolved] (SPARK-12651) mllib deprecation messages mention non-existent version 1.7.0

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12651. --- Resolution: Duplicate > mllib deprecation messages mention non-existent version 1.7.0 >

[jira] [Resolved] (SPARK-3620) Refactor config option handling code for spark-submit

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3620. -- Resolution: Won't Fix I think this is obsolete, or in some cases implemented, given subsequent

[jira] [Created] (SPARK-12676) There is no way to stop a spark-streaming job from a worker in case of errors.

2016-01-06 Thread Sohaib Iftikhar (JIRA)
Sohaib Iftikhar created SPARK-12676: --- Summary: There is no way to stop a spark-streaming job from a worker in case of errors. Key: SPARK-12676 URL: https://issues.apache.org/jira/browse/SPARK-12676

[jira] [Assigned] (SPARK-12542) Support intersect/except in Hive SQL

2016-01-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-12542: -- Assignee: Davies Liu (was: Xiao Li) > Support intersect/except in Hive SQL >

[jira] [Updated] (SPARK-12542) Support intersect/except in Hive SQL

2016-01-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-12542: --- Summary: Support intersect/except in Hive SQL (was: Support union/intersect/except in Hive SQL) >

[jira] [Assigned] (SPARK-12672) Streaming batch ui can't be opened in jobs page in yarn mode.

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12672: Assignee: (was: Apache Spark) > Streaming batch ui can't be opened in jobs page in

[jira] [Commented] (SPARK-12672) Streaming batch ui can't be opened in jobs page in yarn mode.

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085228#comment-15085228 ] Apache Spark commented on SPARK-12672: -- User 'SaintBacchus' has created a pull request for this

[jira] [Assigned] (SPARK-12672) Streaming batch ui can't be opened in jobs page in yarn mode.

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12672: Assignee: Apache Spark > Streaming batch ui can't be opened in jobs page in yarn mode. >

[jira] [Commented] (SPARK-12430) Temporary folders do not get deleted after Task completes causing problems with disk space.

2016-01-06 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085306#comment-15085306 ] Jean-Baptiste Onofré commented on SPARK-12430: -- I just checked in Utils and

[jira] [Updated] (SPARK-12665) Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used

2016-01-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-12665: --- Component/s: GraphX > Remove Vector, VectorSuite and GraphKryoRegistrator which are

[jira] [Resolved] (SPARK-1358) Continuous integrated test should be involved in Spark ecosystem

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-1358. -- Resolution: Won't Fix Did this pre-date amplab Jenkins? there are already a lot of integration tests

[jira] [Resolved] (SPARK-4624) Errors when reading/writtign to S3 large object files

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4624. -- Resolution: Cannot Reproduce This sounds like an S3 issue, but reopen if you can still reproduce and

[jira] [Commented] (SPARK-3665) Java API for GraphX

2016-01-06 Thread Romi Kuntsman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085452#comment-15085452 ] Romi Kuntsman commented on SPARK-3665: -- So at what version of Spark is it expected to happen? > Java

[jira] [Issue Comment Deleted] (SPARK-3665) Java API for GraphX

2016-01-06 Thread Romi Kuntsman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romi Kuntsman updated SPARK-3665: - Comment: was deleted (was: So at what version of Spark is it expected to happen?) > Java API for

[jira] [Updated] (SPARK-12665) Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used

2016-01-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-12665: --- Summary: Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no

[jira] [Updated] (SPARK-12665) Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used

2016-01-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-12665: --- Description: Whole code of Vector.scala, VectorSuite.scala and GraphKryoRegistrator.scala

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Attachment: screenshot-1.png > Prepending base URI of job description is missing >

[jira] [Created] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-12673: --- Summary: Prepending base URI of job description is missing Key: SPARK-12673 URL: https://issues.apache.org/jira/browse/SPARK-12673 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-4408) Behavior difference between spark-submit conf vs cmd line args

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4408. -- Resolution: Not A Problem Yes, I think you've accurately described how it works, but I think that's by

[jira] [Resolved] (SPARK-2867) saveAsHadoopFile() in PairRDDFunction.scala should allow use other OutputCommiter class

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2867. -- Resolution: Not A Problem This can be specified in the Hadoop {{Configuration}} > saveAsHadoopFile()

[jira] [Resolved] (SPARK-3523) GraphX graph partitioning strategy

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3523. -- Resolution: Won't Fix > GraphX graph partitioning strategy > -- > >

[jira] [Resolved] (SPARK-4725) Re-think custom shuffle serializers for vertex messages

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4725. -- Resolution: Won't Fix > Re-think custom shuffle serializers for vertex messages >

[jira] [Resolved] (SPARK-4659) Implement K-core decomposition algorithm

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4659. -- Resolution: Won't Fix > Implement K-core decomposition algorithm >

[jira] [Resolved] (SPARK-4722) StreamingLinearRegression should return a DStream of weights when calling trainOn

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4722. -- Resolution: Won't Fix > StreamingLinearRegression should return a DStream of weights when calling >

[jira] [Commented] (SPARK-12675) Executor dies because of ClassCastException and causes timeout

2016-01-06 Thread Alexandru Rosianu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085414#comment-15085414 ] Alexandru Rosianu commented on SPARK-12675: --- P.S. I hope this is the right place to report the

[jira] [Updated] (SPARK-12675) Executor dies because of ClassCastException and causes timeout

2016-01-06 Thread Alexandru Rosianu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandru Rosianu updated SPARK-12675: -- Summary: Executor dies because of ClassCastException and causes timeout (was: Spark

[jira] [Commented] (SPARK-12628) SparkUI: weird formatting on additional metrics tooltip

2016-01-06 Thread Vijay Kiran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085434#comment-15085434 ] Vijay Kiran commented on SPARK-12628: - I think it is caused by pre-wrapping in css. Not sure why it

[jira] [Commented] (SPARK-3665) Java API for GraphX

2016-01-06 Thread Romi Kuntsman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085454#comment-15085454 ] Romi Kuntsman commented on SPARK-3665: -- So at what version of Spark is it expected to happen? >

[jira] [Updated] (SPARK-12340) overstep the bounds of Int in SparkPlan.executeTake

2016-01-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-12340: --- Assignee: QiangCai (was: Apache Spark) > overstep the bounds of Int in

[jira] [Resolved] (SPARK-12340) overstep the bounds of Int in SparkPlan.executeTake

2016-01-06 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-12340. Resolution: Fixed Fix Version/s: 2.0.0 > overstep the bounds of Int in

[jira] [Resolved] (SPARK-12578) Parser should not silently ignore the distinct keyword used in an aggregate function when OVER clause is used

2016-01-06 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12578. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.0.0 > Parser should

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

2016-01-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Commented] (SPARK-11139) Make SparkContext.stop() exception-safe

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085208#comment-15085208 ] Sean Owen commented on SPARK-11139: --- Yes, but not for the streaming context (see SPARK-11137). A

[jira] [Commented] (SPARK-12621) ArrayIndexOutOfBoundsException when running sqlContext.sql(...)

2016-01-06 Thread Sasi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085192#comment-15085192 ] Sasi commented on SPARK-12621: -- Is there any fix about it on Spark 1.5.2? I don't know if I can do the tests

[jira] [Resolved] (SPARK-4190) Allow users to provide transformation rules at JSON ingest

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4190. -- Resolution: Won't Fix I suspect this is covered by much subsequent work on reading JSON as dataframes.

[jira] [Resolved] (SPARK-4993) execute rdd.count failed when storage level is OFF_HEAP

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4993. -- Resolution: Not A Problem This sounds like a Tachyon client problem, which is either subsequently

[jira] [Updated] (SPARK-12675) Executor dies because of ClassCastException and causes timeout

2016-01-06 Thread Alexandru Rosianu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandru Rosianu updated SPARK-12675: -- Priority: Minor (was: Critical) > Executor dies because of ClassCastException and

[jira] [Commented] (SPARK-12675) Executor dies because of ClassCastException and causes timeout

2016-01-06 Thread Alexandru Rosianu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085495#comment-15085495 ] Alexandru Rosianu commented on SPARK-12675: --- I found a workaround. I still don't know what the

[jira] [Created] (SPARK-12677) Lazy file discovery for parquet

2016-01-06 Thread Tiago Albineli Motta (JIRA)
Tiago Albineli Motta created SPARK-12677: Summary: Lazy file discovery for parquet Key: SPARK-12677 URL: https://issues.apache.org/jira/browse/SPARK-12677 Project: Spark Issue Type:

[jira] [Commented] (SPARK-7481) Add Hadoop 2.6+ profile to pull in object store FS accessors

2016-01-06 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085477#comment-15085477 ] Steve Loughran commented on SPARK-7481: --- Josh, there is a 2.6 profile —but all it currently does is

[jira] [Updated] (SPARK-12640) Add benchmarks to measure the speed ups of UnsafeRowParquetReaderReader.

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12640: -- Priority: Minor (was: Major) Component/s: SQL Issue Type: Task (was: Bug) > Add

[jira] [Updated] (SPARK-12668) Renaming CSV options to be similar to Pandas and R

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12668: -- Fix Version/s: (was: 2.0.0) (Don't set fix version if it's not resolved) > Renaming CSV options

[jira] [Updated] (SPARK-12621) ArrayIndexOutOfBoundsException when running sqlContext.sql(...)

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12621: -- Component/s: SQL > ArrayIndexOutOfBoundsException when running sqlContext.sql(...) >

[jira] [Updated] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12650: -- Component/s: Spark Submit > No means to specify Xmx settings for SparkSubmit in yarn-cluster mode >

[jira] [Updated] (SPARK-12438) Add SQLUserDefinedType support for encoder

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12438: -- Assignee: Liang-Chi Hsieh > Add SQLUserDefinedType support for encoder >

[jira] [Commented] (SPARK-12340) overstep the bounds of Int in SparkPlan.executeTake

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085726#comment-15085726 ] Apache Spark commented on SPARK-12340: -- User 'QiangCai' has created a pull request for this issue:

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085734#comment-15085734 ] Sean Owen commented on SPARK-12650: --- Hm, the default heap size in the JVM isn't 8GB is it? I just

[jira] [Resolved] (SPARK-12676) There is no way to stop a spark-streaming job from a worker in case of errors.

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12676. --- Resolution: Not A Problem Your driver app will already fail -- if you make it fail when you want it

[jira] [Commented] (SPARK-11888) Model export/import for spark.ml: DecisionTreeClassifier,Regressor

2016-01-06 Thread Rares Mirica (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085670#comment-15085670 ] Rares Mirica commented on SPARK-11888: -- Is there any chance this will be released in another minor

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085928#comment-15085928 ] Marcelo Vanzin commented on SPARK-12650: You can find the default Xmx like this: {code} java

[jira] [Resolved] (SPARK-12665) Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used

2016-01-06 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12665. - Resolution: Fixed Assignee: Kousuke Saruta Fix Version/s: 2.0.0 > Remove Vector,

[jira] [Commented] (SPARK-9313) Enable a "docker run" invocation in place of PYSPARK_PYTHON

2016-01-06 Thread thom neale (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085990#comment-15085990 ] thom neale commented on SPARK-9313: --- [~joshrosen] There's only one reason I know of so far-- In

[jira] [Commented] (SPARK-12617) socket descriptor leak killing streaming app

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086008#comment-15086008 ] Apache Spark commented on SPARK-12617: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086023#comment-15086023 ] Marcelo Vanzin commented on SPARK-12650: Also, can you clarify this statement: {quote} This

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086029#comment-15086029 ] Marcelo Vanzin commented on SPARK-12650: And one last comment: you can set either the

[jira] [Resolved] (SPARK-11878) Eliminate distribute by in case group by is present with exactly the same grouping expressions

2016-01-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-11878. -- Resolution: Fixed Assignee: Yash Datta Fix Version/s: 2.0.0 >

[jira] [Resolved] (SPARK-7675) PySpark spark.ml Params type conversions

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-7675. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 9581

[jira] [Assigned] (SPARK-12542) Support intersect/except in Hive SQL

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12542: Assignee: Davies Liu (was: Apache Spark) > Support intersect/except in Hive SQL >

[jira] [Commented] (SPARK-12542) Support intersect/except in Hive SQL

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086042#comment-15086042 ] Apache Spark commented on SPARK-12542: -- User 'davies' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12542) Support intersect/except in Hive SQL

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12542: Assignee: Apache Spark (was: Davies Liu) > Support intersect/except in Hive SQL >

[jira] [Updated] (SPARK-11531) PySpark SparseVector: improve error message for bad indices

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-11531: -- Summary: PySpark SparseVector: improve error message for bad indices (was: PySpark

[jira] [Updated] (SPARK-11531) PySpark SparseVector: improve error message for bad indices

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-11531: -- Assignee: Rekha Joshi > PySpark SparseVector: improve error message for bad indices >

[jira] [Resolved] (SPARK-11531) PySpark SparseVector: improve error message for bad indices

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-11531. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 9525

[jira] [Assigned] (SPARK-9716) BinaryClassificationEvaluator should accept Double prediction column

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9716: --- Assignee: Apache Spark > BinaryClassificationEvaluator should accept Double prediction

[jira] [Assigned] (SPARK-9716) BinaryClassificationEvaluator should accept Double prediction column

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9716: --- Assignee: (was: Apache Spark) > BinaryClassificationEvaluator should accept Double

[jira] [Resolved] (SPARK-11945) Add computeCost to KMeansModel for PySpark spark.ml

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-11945. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 9931

[jira] [Resolved] (SPARK-11815) PySpark DecisionTreeClassifier & DecisionTreeRegressor should support setSeed

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-11815. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 9807

[jira] [Commented] (SPARK-12662) Add a local sort operator to DataFrame used by randomSplit

2016-01-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086056#comment-15086056 ] Davies Liu commented on SPARK-12662: Another way to make the DataFrame deterministic is materialize

[jira] [Updated] (SPARK-10809) Single-document topicDistributions method for LocalLDAModel

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10809: -- Shepherd: Joseph K. Bradley Target Version/s: 2.0.0 > Single-document

[jira] [Updated] (SPARK-10809) Single-document topicDistributions method for LocalLDAModel

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-10809: -- Assignee: yuhao yang > Single-document topicDistributions method for LocalLDAModel >

[jira] [Commented] (SPARK-4819) Remove Guava's "Optional" from public API

2016-01-06 Thread Markus Weimer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086083#comment-15086083 ] Markus Weimer commented on SPARK-4819: -- Over in REEF, we just backportd Java 8's [[Optional]] class.

[jira] [Comment Edited] (SPARK-4819) Remove Guava's "Optional" from public API

2016-01-06 Thread Markus Weimer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086083#comment-15086083 ] Markus Weimer edited comment on SPARK-4819 at 1/6/16 7:09 PM: -- Over in REEF,

[jira] [Created] (SPARK-12678) MapPartitionsRDD

2016-01-06 Thread Guillaume Poulin (JIRA)
Guillaume Poulin created SPARK-12678: Summary: MapPartitionsRDD Key: SPARK-12678 URL: https://issues.apache.org/jira/browse/SPARK-12678 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-4819) Remove Guava's "Optional" from public API

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086092#comment-15086092 ] Sean Owen commented on SPARK-4819: -- That's basically what I did, and then added some Guava API methods

[jira] [Updated] (SPARK-12006) GaussianMixture.train crashes if an initial model is not None

2016-01-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-12006: -- Assignee: Maciej Szymkiewicz > GaussianMixture.train crashes if an initial model is

[jira] [Resolved] (SPARK-12573) Add acknowledge that the parser was initially from Hive

2016-01-06 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12573. - Resolution: Fixed Fix Version/s: 2.0.0 > Add acknowledge that the parser was initially

[jira] [Resolved] (SPARK-12574) Move parser from hive module to catalyst (or sql-core) module

2016-01-06 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12574. - Resolution: Fixed Assignee: Herman van Hovell Fix Version/s: 2.0.0 > Move parser

[jira] [Commented] (SPARK-12678) MapPartitionsRDD

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086126#comment-15086126 ] Apache Spark commented on SPARK-12678: -- User 'gpoulin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12678) MapPartitionsRDD

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12678: Assignee: (was: Apache Spark) > MapPartitionsRDD > > >

[jira] [Assigned] (SPARK-12678) MapPartitionsRDD

2016-01-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12678: Assignee: Apache Spark > MapPartitionsRDD > > > Key:

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread John Vines (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086133#comment-15086133 ] John Vines commented on SPARK-12650: In the test example I was using, I set driver and executor to

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread John Vines (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086135#comment-15086135 ] John Vines commented on SPARK-12650: {code}[root@datanode1-systemtest-john-1 /]# java

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread John Vines (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086138#comment-15086138 ] John Vines commented on SPARK-12650: I'm launching the spark job from inside an App Master, as I said

[jira] [Updated] (SPARK-12678) MapPartitionsRDD should clear reference to prev RDD

2016-01-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12678: -- Summary: MapPartitionsRDD should clear reference to prev RDD (was: MapPartitionsRDD) >

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

2016-01-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086148#comment-15086148 ] Marcelo Vanzin commented on SPARK-12650: Do you need the launcher process around after it's

[jira] [Updated] (SPARK-12662) Add a local sort operator to DataFrame used by randomSplit

2016-01-06 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12662: - Description: With {{./bin/spark-shell --master=local-cluster[2,1,2014]}}, the following code will

[jira] [Updated] (SPARK-12662) Add a local sort operator to DataFrame used by randomSplit

2016-01-06 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12662: - Summary: Add a local sort operator to DataFrame used by randomSplit (was: Add document to randomSplit

[jira] [Commented] (SPARK-12662) Add a local sort operator to DataFrame used by randomSplit

2016-01-06 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085764#comment-15085764 ] Yin Huai commented on SPARK-12662: -- btw, with local sort operator, we can make row ordering in a

  1   2   3   >