[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster

2015-02-20 Thread Florian Verhein (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329940#comment-14329940 ] Florian Verhein commented on SPARK-5629: Agree. My point was more about avoiding

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329885#comment-14329885 ] Zhan Zhang commented on SPARK-1537: --- [~vanzin] We should centralized all comments and r

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329854#comment-14329854 ] Kay Ousterhout commented on SPARK-5928: --- One more question: are you sure this is a p

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329852#comment-14329852 ] Marcelo Vanzin commented on SPARK-1537: --- Hi [~zhzhan], bq. But It is hard to commen

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329851#comment-14329851 ] Kay Ousterhout commented on SPARK-5928: --- Is it possible this is caused because the s

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329828#comment-14329828 ] Zhan Zhang commented on SPARK-1537: --- [~sowen] In JIRA, we share the code so that other p

[jira] [Commented] (SPARK-5937) [YARN] ClientSuite must set YARN mode to true to ensure correct SparkHadoopUtil implementation is used.

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329814#comment-14329814 ] Apache Spark commented on SPARK-5937: - User 'harishreedharan' has created a pull reque

[jira] [Updated] (SPARK-2336) Approximate k-NN Models for MLLib

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2336: - Target Version/s: 1.4.0 > Approximate k-NN Models for MLLib > - >

[jira] [Updated] (SPARK-2335) k-Nearest Neighbor classification and regression for MLLib

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2335: - Target Version/s: 1.4.0 > k-Nearest Neighbor classification and regression for MLLib > ---

[jira] [Created] (SPARK-5937) [YARN] ClientSuite must set YARN mode to true to ensure correct SparkHadoopUtil implementation is used.

2015-02-20 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created SPARK-5937: --- Summary: [YARN] ClientSuite must set YARN mode to true to ensure correct SparkHadoopUtil implementation is used. Key: SPARK-5937 URL: https://issues.apache.org/jira/browse/S

[jira] [Created] (SPARK-5936) Automatically convert a StructType to a MapType when the number of fields exceed a threshold.

2015-02-20 Thread Yin Huai (JIRA)
Yin Huai created SPARK-5936: --- Summary: Automatically convert a StructType to a MapType when the number of fields exceed a threshold. Key: SPARK-5936 URL: https://issues.apache.org/jira/browse/SPARK-5936 Pro

[jira] [Commented] (SPARK-5935) Accept MapType in the schema provided to a JSON dataset.

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329803#comment-14329803 ] Apache Spark commented on SPARK-5935: - User 'yhuai' has created a pull request for thi

[jira] [Created] (SPARK-5935) Accept MapType in the schema provided to a JSON dataset.

2015-02-20 Thread Yin Huai (JIRA)
Yin Huai created SPARK-5935: --- Summary: Accept MapType in the schema provided to a JSON dataset. Key: SPARK-5935 URL: https://issues.apache.org/jira/browse/SPARK-5935 Project: Spark Issue Type: Sub-

[jira] [Updated] (SPARK-2138) The KMeans algorithm in the MLlib can lead to the Serialized Task size become bigger and bigger

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2138: - Target Version/s: 1.4.0 > The KMeans algorithm in the MLlib can lead to the Serialized Task size b

[jira] [Closed] (SPARK-1892) Add an OWL-QN optimizer for L1 regularized optimizations.

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-1892. Resolution: Duplicate > Add an OWL-QN optimizer for L1 regularized optimizations. >

[jira] [Updated] (SPARK-1856) Standardize MLlib interfaces

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1856: - Issue Type: Umbrella (was: New Feature) > Standardize MLlib interfaces >

[jira] [Closed] (SPARK-1794) Generic ADMM implementation for SVM, lasso, and L1-regularized logistic regression

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-1794. Resolution: Duplicate > Generic ADMM implementation for SVM, lasso, and L1-regularized logistic > r

[jira] [Closed] (SPARK-1673) GLMNET implementation in Spark

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-1673. Resolution: Duplicate > GLMNET implementation in Spark > -- > >

[jira] [Updated] (SPARK-1655) In naive Bayes, store conditional probabilities distributively.

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1655: - Target Version/s: 1.4.0 > In naive Bayes, store conditional probabilities distributively. > --

[jira] [Closed] (SPARK-1418) Python MLlib's _get_unmangled_rdd should uncache RDDs when training is done

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-1418. Resolution: Implemented Fix Version/s: 1.2.0 > Python MLlib's _get_unmangled_rdd should uncac

[jira] [Updated] (SPARK-1359) SGD implementation is not efficient

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1359: - Target Version/s: 1.4.0 > SGD implementation is not efficient > --

[jira] [Created] (SPARK-5934) DStreamGraph.clearMetadata attempts to unpersist the same RDD multiple times

2015-02-20 Thread Nick Pritchard (JIRA)
Nick Pritchard created SPARK-5934: - Summary: DStreamGraph.clearMetadata attempts to unpersist the same RDD multiple times Key: SPARK-5934 URL: https://issues.apache.org/jira/browse/SPARK-5934 Project:

[jira] [Closed] (SPARK-1014) MultilogisticRegressionWithSGD

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-1014. Resolution: Duplicate We support multinomial logistic regression with LBFGS in 1.3. I marked this J

[jira] [Updated] (SPARK-5888) Add OneHotEncoder as a Transformer

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-5888: - Summary: Add OneHotEncoder as a Transformer (was: Add OneHotEncoder) > Add OneHotEncoder as a Tra

[jira] [Updated] (SPARK-1473) Feature selection for high dimensional datasets

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1473: - Issue Type: Umbrella (was: New Feature) > Feature selection for high dimensional datasets > -

[jira] [Commented] (SPARK-5912) Programming guide for feature selection

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329773#comment-14329773 ] Apache Spark commented on SPARK-5912: - User 'avulanov' has created a pull request for

[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster

2015-02-20 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329771#comment-14329771 ] Nicholas Chammas commented on SPARK-5629: - [~florianverhein] - Hmm... Thinking abo

[jira] [Resolved] (SPARK-5896) toDF in python doesn't work with tuple/list w/o names

2015-02-20 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5896. - Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 4679 [https:/

[jira] [Resolved] (SPARK-5898) Can't create DataFrame from Pandas data frame

2015-02-20 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-5898. - Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 4679 [https:/

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329739#comment-14329739 ] Sean Owen commented on SPARK-1537: -- [~zzhan] You have provided a patch as a PR right? any

[jira] [Issue Comment Deleted] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated SPARK-1537: -- Comment: was deleted (was: [~sowen] By the way, I am not waiting for someone to give me the patch. It i

[jira] [Updated] (SPARK-4081) Categorical feature indexing

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4081: - Target Version/s: 1.4.0 (was: 1.2.0) > Categorical feature indexing > ---

[jira] [Updated] (SPARK-3249) Fix links in ScalaDoc that cause warning messages in `sbt/sbt unidoc`

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3249: - Target Version/s: (was: 1.2.0) > Fix links in ScalaDoc that cause warning messages in `sbt/sbt u

[jira] [Commented] (SPARK-5516) ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.actor.default-dispatcher-22] shutting down ActorSystem [sparkDriver] java.lang.OutOfMemoryError: Jav

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329718#comment-14329718 ] Xiangrui Meng commented on SPARK-5516: -- [~wuyukai] Could you provide all the paramete

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329720#comment-14329720 ] Imran Rashid commented on SPARK-5928: - Actually, there *is* some weirdness in how spar

[jira] [Updated] (SPARK-5516) ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.actor.default-dispatcher-22] shutting down ActorSystem [sparkDriver] java.lang.OutOfMemoryError: Java

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-5516: - Fix Version/s: (was: 1.2.2) > ActorSystemImpl: Uncaught fatal error from thread > [sparkDrive

[jira] [Updated] (SPARK-5516) ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.actor.default-dispatcher-22] shutting down ActorSystem [sparkDriver] java.lang.OutOfMemoryError: Java

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-5516: - Target Version/s: 1.4.0 (was: 1.2.0) > ActorSystemImpl: Uncaught fatal error from thread > [spar

[jira] [Updated] (SPARK-4406) SVD should check for k < 1

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4406: - Target Version/s: 1.3.0 (was: 1.2.0) > SVD should check for k < 1 > -- >

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329704#comment-14329704 ] Zhan Zhang commented on SPARK-1537: --- [~sowen] By the way, I am not waiting for someone t

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329700#comment-14329700 ] Zhan Zhang commented on SPARK-1537: --- [~sowen] From the whole context, I believe you unde

[jira] [Commented] (SPARK-5912) Programming guide for feature selection

2015-02-20 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329692#comment-14329692 ] Joseph K. Bradley commented on SPARK-5912: -- Great, thanks! I build and view them

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329691#comment-14329691 ] Sean Owen commented on SPARK-1537: -- [~zzhan] I also can't figure out what you are suggest

[jira] [Commented] (SPARK-5912) Programming guide for feature selection

2015-02-20 Thread Alexander Ulanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329685#comment-14329685 ] Alexander Ulanov commented on SPARK-5912: - I've almost written the ChiSquared sect

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329681#comment-14329681 ] Marcelo Vanzin commented on SPARK-1537: --- It's impossible to submit a patch when the

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329678#comment-14329678 ] Zhan Zhang commented on SPARK-1537: --- [~vanzin] I declare "integrate your code" from the

[jira] [Updated] (SPARK-5933) Centralize deprecated configs in SparkConf

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5933: - Description: Deprecated configs are currently all strewn across the code base. It would be good to simplif

[jira] [Created] (SPARK-5933) Centralize deprecated configs in SparkConf

2015-02-20 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5933: Summary: Centralize deprecated configs in SparkConf Key: SPARK-5933 URL: https://issues.apache.org/jira/browse/SPARK-5933 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-5932) Use consistent naming for byte properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5932: - Description: This is SPARK-5931's sister issue. The naming of existing byte configs is inconsistent. We c

[jira] [Updated] (SPARK-5931) Use consistent naming for time properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5931: - Description: This is SPARK-5932's sister issue. The naming of existing time configs is inconsistent. We c

[jira] [Updated] (SPARK-5931) Use consistent naming for time properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5931: - Description: This is SPARK-5932's sister issue. The naming of existing time configs is inconsistent. We c

[jira] [Updated] (SPARK-5932) Use consistent naming for byte properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5932: - Description: This is SPARK-5931's sister issue. The naming of existing byte configs is inconsistent. We c

[jira] [Created] (SPARK-5932) Use consistent naming for byte properties

2015-02-20 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5932: Summary: Use consistent naming for byte properties Key: SPARK-5932 URL: https://issues.apache.org/jira/browse/SPARK-5932 Project: Spark Issue Type: Bug Com

[jira] [Updated] (SPARK-5932) Use consistent naming for byte properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5932: - Description: This is SPARK-5931's sister issue. The naming of existing byte configs is inconsistent. We c

[jira] [Updated] (SPARK-5931) Use consistent naming for time properties

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5931: - Description: This is SPARK-5932's sister issue. The naming of existing time configs is inconsistent. We c

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329664#comment-14329664 ] Zhan Zhang commented on SPARK-1537: --- [~vanzin] If you don't have bandwidth, or don't kno

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329660#comment-14329660 ] Marcelo Vanzin commented on SPARK-1537: --- Hi [~zzhan], I already posted the link to

[jira] [Created] (SPARK-5931) Use consistent naming for time properties

2015-02-20 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5931: Summary: Use consistent naming for time properties Key: SPARK-5931 URL: https://issues.apache.org/jira/browse/SPARK-5931 Project: Spark Issue Type: Bug Com

[jira] [Comment Edited] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329649#comment-14329649 ] Zhan Zhang edited comment on SPARK-1537 at 2/20/15 10:14 PM: -

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329649#comment-14329649 ] Zhan Zhang commented on SPARK-1537: --- [~vanzin] Thanks for the comments. I don't unders

[jira] [Commented] (SPARK-3368) Spark cannot be used with Avro and Parquet

2015-02-20 Thread Daniel Fry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329645#comment-14329645 ] Daniel Fry commented on SPARK-3368: --- Hey fwiw I encountered this recently with spark 1.1

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Description: The description makes it sound like the retryWait itself defaults to 15 seconds, when it's a

[jira] [Commented] (SPARK-4655) Split Stage into ShuffleMapStage and ResultStage subclasses

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329641#comment-14329641 ] Apache Spark commented on SPARK-4655: - User 'ilganeli' has created a pull request for

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Affects Version/s: 1.2.0 > Documented default of spark.shuffle.io.retryWait is confusing > ---

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Priority: Trivial (was: Minor) > Documented default of spark.shuffle.io.retryWait is confusing >

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Summary: Documented default of spark.shuffle.io.retryWait is confusing (was: Documented default of spark.

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is not consistent

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Priority: Minor (was: Major) > Documented default of spark.shuffle.io.retryWait is not consistent > -

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Description: The description makes it sound like the retryWait itself defaults to 15 seconds, when it's a

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is confusing.

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Summary: Documented default of spark.shuffle.io.retryWait is confusing. (was: Documented default of spark

[jira] [Updated] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is not consistent

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5930: - Description: 5 != 15: {code} spark.shuffle.io.retryWait 5 (Netty only) Seconds to wait between

[jira] [Created] (SPARK-5930) Documented default of spark.shuffle.io.retryWait is not consistent

2015-02-20 Thread Andrew Or (JIRA)
Andrew Or created SPARK-5930: Summary: Documented default of spark.shuffle.io.retryWait is not consistent Key: SPARK-5930 URL: https://issues.apache.org/jira/browse/SPARK-5930 Project: Spark Iss

[jira] [Commented] (SPARK-5281) Registering table on RDD is giving MissingRequirementError

2015-02-20 Thread Sebastian YEPES FERNANDEZ (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329628#comment-14329628 ] Sebastian YEPES FERNANDEZ commented on SPARK-5281: -- Also having this issu

[jira] [Comment Edited] (SPARK-5281) Registering table on RDD is giving MissingRequirementError

2015-02-20 Thread Sebastian YEPES FERNANDEZ (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329628#comment-14329628 ] Sebastian YEPES FERNANDEZ edited comment on SPARK-5281 at 2/20/15 9:54 PM: -

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329623#comment-14329623 ] Imran Rashid commented on SPARK-5928: - sometimes this also results in exceptions like

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329613#comment-14329613 ] Imran Rashid commented on SPARK-1391: - Here is a minimal program to demonstrate the pr

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329605#comment-14329605 ] Imran Rashid commented on SPARK-1391: - [~coderplay], I assume you are no longer lookin

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329584#comment-14329584 ] Imran Rashid commented on SPARK-5928: - Here are some thoughts on we *might* fix this.

[jira] [Updated] (SPARK-4705) Driver retries in cluster mode always fail if event logging is enabled

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4705: - Summary: Driver retries in cluster mode always fail if event logging is enabled (was: Driver retries in y

[jira] [Commented] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329568#comment-14329568 ] Imran Rashid commented on SPARK-5928: - (just edited the description -- I mistakenly th

[jira] [Updated] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-5928: Description: If a shuffle block is over 2GB, the shuffle fails, with an uninformative exception. T

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

2015-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329554#comment-14329554 ] Imran Rashid commented on SPARK-1476: - I spent a little time with [~sandyr] on this to

[jira] [Created] (SPARK-5929) Pyspark: Register a pip requirements file with spark_context

2015-02-20 Thread Buck (JIRA)
Buck created SPARK-5929: --- Summary: Pyspark: Register a pip requirements file with spark_context Key: SPARK-5929 URL: https://issues.apache.org/jira/browse/SPARK-5929 Project: Spark Issue Type: Improve

[jira] [Created] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2015-02-20 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-5928: --- Summary: Remote Shuffle Blocks cannot be more than 2 GB Key: SPARK-5928 URL: https://issues.apache.org/jira/browse/SPARK-5928 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2015-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329460#comment-14329460 ] Marcelo Vanzin commented on SPARK-1537: --- Hi [~zzhan], thanks for uploading the docum

[jira] [Commented] (SPARK-5918) Spark Thrift server reports metadata for VARCHAR column as STRING in result set schema

2015-02-20 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329417#comment-14329417 ] Michael Armbrust commented on SPARK-5918: - This was a conscious design decision si

[jira] [Updated] (SPARK-5095) Support launching multiple mesos executors in coarse grained mesos mode

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5095: - Assignee: Timothy Chen > Support launching multiple mesos executors in coarse grained mesos mode > ---

[jira] [Updated] (SPARK-5095) Support launching multiple mesos executors in coarse grained mesos mode

2015-02-20 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-5095: - Affects Version/s: 1.0.0 > Support launching multiple mesos executors in coarse grained mesos mode > -

[jira] [Commented] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329329#comment-14329329 ] Apache Spark commented on SPARK-5926: - User 'yanboliang' has created a pull request fo

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:34 PM: -

[jira] [Commented] (SPARK-5927) Modify FPGrowth's partition strategy to reduce transactions in partitions

2015-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329306#comment-14329306 ] Apache Spark commented on SPARK-5927: - User 'viirya' has created a pull request for th

[jira] [Commented] (SPARK-5925) YARN - Spark progress bar stucks at 10% but after finishing shows 100%

2015-02-20 Thread Laszlo Fesus (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329303#comment-14329303 ] Laszlo Fesus commented on SPARK-5925: - Yes, but I thought it would be quite useful if

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:14 PM: -

[jira] [Created] (SPARK-5927) Modify FPGrowth's partition strategy to reduce transactions in partitions

2015-02-20 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-5927: -- Summary: Modify FPGrowth's partition strategy to reduce transactions in partitions Key: SPARK-5927 URL: https://issues.apache.org/jira/browse/SPARK-5927 Project:

[jira] [Commented] (SPARK-5832) Add Affinity Propagation clustering algorithm

2015-02-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329284#comment-14329284 ] Liang-Chi Hsieh commented on SPARK-5832: The time complexity O(nnz * K) is just fo

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:06 PM: -

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:03 PM: -

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329251#comment-14329251 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:01 PM: -

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329251#comment-14329251 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 5:59 PM: -

[jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329251#comment-14329251 ] Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:01 PM: -

[jira] [Commented] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command

2015-02-20 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269 ] Yanbo Liang commented on SPARK-5926: This is because that in DataFrameImpl {code:titl

[jira] [Updated] (SPARK-5832) Add Affinity Propagation clustering algorithm

2015-02-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-5832: - Target Version/s: 1.4.0 > Add Affinity Propagation clustering algorithm >

  1   2   >