[jira] [Assigned] (SPARK-3212) Improve the clarity of caching semantics

2014-08-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-3212: --- Assignee: Michael Armbrust > Improve the clarity of caching semantics > -

[jira] [Created] (SPARK-3295) [Spark SQL] schemaRdd1 ++ schemaRdd2 does not return another SchemaRdd

2014-08-28 Thread Evan Chan (JIRA)
Evan Chan created SPARK-3295: Summary: [Spark SQL] schemaRdd1 ++ schemaRdd2 does not return another SchemaRdd Key: SPARK-3295 URL: https://issues.apache.org/jira/browse/SPARK-3295 Project: Spark

[jira] [Resolved] (SPARK-3279) Remove useless field variable in ApplicationMaster

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3279. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2177 [https://github.com

[jira] [Updated] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-08-28 Thread guowei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guowei updated SPARK-3292: -- Description: such as repartition groupby join and cogroup for example. if i want the shuffle outputs save as h

[jira] [Updated] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-08-28 Thread guowei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guowei updated SPARK-3292: -- Description: such as repartition groupby join and cogroup it's too expensive , for example. if i want the shuf

[jira] [Comment Edited] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Colin B. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114881#comment-14114881 ] Colin B. edited comment on SPARK-3266 at 8/29/14 5:38 AM: -- Broken

[jira] [Updated] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-08-28 Thread guowei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guowei updated SPARK-3292: -- Description: such as repartition groupby join and cogroup it's too expensive , for example if i want outputs s

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Colin B. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114881#comment-14114881 ] Colin B. commented on SPARK-3266: - Broken for JavaDoubleRDD: fold, reduce, min, max {code}

[jira] [Updated] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-08-28 Thread guowei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guowei updated SPARK-3292: -- Summary: Shuffle Tasks run incessantly even though there's no inputs (was: Shuffle Tasks run indefinitely even

[jira] [Commented] (SPARK-3280) Made sort-based shuffle the default implementation

2014-08-28 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114873#comment-14114873 ] Burak Yavuz commented on SPARK-3280: I don't have as detailed a comparison like Josh h

[jira] [Updated] (SPARK-3280) Made sort-based shuffle the default implementation

2014-08-28 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-3280: --- Attachment: hash-sort-comp.png > Made sort-based shuffle the default implementation > ---

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114870#comment-14114870 ] Josh Rosen commented on SPARK-3266: --- [~pwendell] I think the JavaRDDLike trait compiles

[jira] [Updated] (SPARK-3294) Avoid boxing/unboxing when handling in-memory columnar storage

2014-08-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-3294: Assignee: Cheng Lian > Avoid boxing/unboxing when handling in-memory columnar storage > ---

[jira] [Updated] (SPARK-2219) AddJar doesn't work

2014-08-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2219: Assignee: Cheng Lian (was: Michael Armbrust) > AddJar doesn't work > --- >

[jira] [Updated] (SPARK-2973) Add a way to show tables without executing a job

2014-08-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2973: Assignee: Cheng Lian > Add a way to show tables without executing a job > -

[jira] [Commented] (SPARK-3294) Avoid boxing/unboxing when handling in-memory columnar storage

2014-08-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114836#comment-14114836 ] Cheng Lian commented on SPARK-3294: --- [~marmbrus], would you mind to assign this issue to

[jira] [Created] (SPARK-3294) Avoid boxing/unboxing when handling in-memory columnar storage

2014-08-28 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-3294: - Summary: Avoid boxing/unboxing when handling in-memory columnar storage Key: SPARK-3294 URL: https://issues.apache.org/jira/browse/SPARK-3294 Project: Spark Issue

[jira] [Commented] (SPARK-3292) Shuffle Tasks run indefinitely even though there's no inputs

2014-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114829#comment-14114829 ] Sean Owen commented on SPARK-3292: -- Can you elaborate this? it's not clear whether you're

[jira] [Created] (SPARK-3293) yarn's web show "SUCCEEDED" when the driver throw a exception in yarn-client

2014-08-28 Thread wangfei (JIRA)
wangfei created SPARK-3293: -- Summary: yarn's web show "SUCCEEDED" when the driver throw a exception in yarn-client Key: SPARK-3293 URL: https://issues.apache.org/jira/browse/SPARK-3293 Project: Spark

[jira] [Commented] (SPARK-3291) TestcaseName in createQueryTest should not contain ":"

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114807#comment-14114807 ] Apache Spark commented on SPARK-3291: - User 'chouqin' has created a pull request for t

[jira] [Created] (SPARK-3292) Shuffle Tasks run indefinitely even though there's no inputs

2014-08-28 Thread guowei (JIRA)
guowei created SPARK-3292: - Summary: Shuffle Tasks run indefinitely even though there's no inputs Key: SPARK-3292 URL: https://issues.apache.org/jira/browse/SPARK-3292 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-3250) More Efficient Sampling

2014-08-28 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114789#comment-14114789 ] Erik Erlandson commented on SPARK-3250: --- I did some experiments with sampling that m

[jira] [Created] (SPARK-3291) TestcaseName in createQueryTest should not contain ":"

2014-08-28 Thread Qiping Li (JIRA)
Qiping Li created SPARK-3291: Summary: TestcaseName in createQueryTest should not contain ":" Key: SPARK-3291 URL: https://issues.apache.org/jira/browse/SPARK-3291 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3272) Calculate prediction for nodes separately from calculating information gain for splits in decision tree

2014-08-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114742#comment-14114742 ] Joseph K. Bradley commented on SPARK-3272: -- Hi Qiping, you are right; I missed th

[jira] [Commented] (SPARK-2594) Add CACHE TABLE AS SELECT ...

2014-08-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114729#comment-14114729 ] Michael Armbrust commented on SPARK-2594: - Its a lot of overhead to assign issues

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3288: - Affects Version/s: 1.1.0 > All fields in TaskMetrics should be private and use getters/setters >

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3277: - Fix Version/s: 1.1.0 > LZ4 compression cause the the ExternalSort exception > ---

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3277: - Affects Version/s: (was: 1.2.0) > LZ4 compression cause the the ExternalSort exception >

[jira] [Updated] (SPARK-3234) SPARK_HADOOP_VERSION doesn't have a valid value by default in make-distribution.sh

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3234: --- Target Version/s: 1.2.0 > SPARK_HADOOP_VERSION doesn't have a valid value by default in > ma

[jira] [Resolved] (SPARK-3245) spark insert into hbase class not serialize

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3245. Resolution: Invalid I'm closing this for now because we typically reported only isolated is

[jira] [Updated] (SPARK-3200) Class defined with reference to external variables crashes in REPL.

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3200: --- Component/s: Spark Shell > Class defined with reference to external variables crashes in REPL

[jira] [Updated] (SPARK-2636) Expose job ID in JobWaiter API

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2636: --- Summary: Expose job ID in JobWaiter API (was: no where to get job identifier while submit sp

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114680#comment-14114680 ] Patrick Wendell commented on SPARK-3266: [~joshrosen] is there a solution here tha

[jira] [Commented] (SPARK-2961) Use statistics to skip partitions when reading from in-memory columnar data

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114668#comment-14114668 ] Apache Spark commented on SPARK-2961: - User 'liancheng' has created a pull request for

[jira] [Created] (SPARK-3290) No unpersist callls in SVDPlusPlus

2014-08-28 Thread Dou Wenjuan (JIRA)
Dou Wenjuan created SPARK-3290: -- Summary: No unpersist callls in SVDPlusPlus Key: SPARK-3290 URL: https://issues.apache.org/jira/browse/SPARK-3290 Project: Spark Issue Type: Bug Compon

[jira] [Resolved] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-2970. --- Resolution: Fixed > spark-sql script ends with IOException when EventLogging is enabled > ---

[jira] [Resolved] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3277. Resolution: Fixed Fixed by https://github.com/apache/spark/pull/2187 Thanks to everyone wh

[jira] [Commented] (SPARK-3287) When ResourceManager High Availability is enabled, ApplicationMaster webUI is not displayed.

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114612#comment-14114612 ] Benoy Antony commented on SPARK-3287: - I'll submit a git pull request. > When Resourc

[jira] [Commented] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114614#comment-14114614 ] Benoy Antony commented on SPARK-3286: - I'll submit a git pull request. > Cannot view

[jira] [Commented] (SPARK-3289) Avoid job failures due to rescheduling of failing tasks on buggy machines

2014-08-28 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114611#comment-14114611 ] Mark Hamstra commented on SPARK-3289: - https://github.com/apache/spark/pull/1360 > Av

[jira] [Commented] (SPARK-3272) Calculate prediction for nodes separately from calculating information gain for splits in decision tree

2014-08-28 Thread Qiping Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114591#comment-14114591 ] Qiping Li commented on SPARK-3272: -- Hi Joseph, thanks for your comment, I think checking

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114551#comment-14114551 ] Apache Spark commented on SPARK-3277: - User 'andrewor14' has created a pull request fo

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114484#comment-14114484 ] Mridul Muralidharan commented on SPARK-3277: Sounds great, thx ! I suspect it

[jira] [Updated] (SPARK-3289) Avoid job failures due to rescheduling of failing tasks on buggy machines

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3289: -- Summary: Avoid job failures due to rescheduling of failing tasks on buggy machines (was: Prevent compl

[jira] [Created] (SPARK-3289) Prevent complete job failures due to rescheduling of failing tasks on buggy machines

2014-08-28 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-3289: - Summary: Prevent complete job failures due to rescheduling of failing tasks on buggy machines Key: SPARK-3289 URL: https://issues.apache.org/jira/browse/SPARK-3289 Project:

[jira] [Resolved] (SPARK-3190) Creation of large graph(> 2.15 B nodes) seems to be broken:possible overflow somewhere

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3190. --- Resolution: Fixed Fix Version/s: 1.0.3 1.1.1 1.2.0 Issue

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114453#comment-14114453 ] Apache Spark commented on SPARK-3266: - User 'JoshRosen' has created a pull request for

[jira] [Updated] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3286: Attachment: (was: SPARK-3286.patch) > Cannot view ApplicationMaster UI when Yarn’s url scheme i

[jira] [Updated] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3286: Attachment: SPARK-3286-branch-1-0.patch > Cannot view ApplicationMaster UI when Yarn’s url scheme i

[jira] [Updated] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3286: Attachment: SPARK-3286.patch Attaching the patch for the master > Cannot view ApplicationMaster UI

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3277: --- Description: I tested the LZ4 compression,and it come up with such problem.(with wordcount) Also I te

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3277: --- Assignee: Andrew Or > LZ4 compression cause the the ExternalSort exception >

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3266: -- Affects Version/s: 1.1.0 JavaRDDLike probably should be an abstract class. I think the current trait

[jira] [Commented] (SPARK-3260) Yarn - pass acls along with executor launch

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114408#comment-14114408 ] Apache Spark commented on SPARK-3260: - User 'tgravescs' has created a pull request for

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Amey Chaugule (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114413#comment-14114413 ] Amey Chaugule commented on SPARK-3266: -- No worries, I initially assumed my runtime en

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114391#comment-14114391 ] Sean Owen commented on SPARK-3266: -- (Mea culpa! The example shows this is a legitimate qu

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Colin B. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114387#comment-14114387 ] Colin B. commented on SPARK-3266: - So there is no method: {code} org.apache.spark.api.java

[jira] [Commented] (SPARK-3183) Add option for requesting full YARN cluster

2014-08-28 Thread Shay Rojansky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114368#comment-14114368 ] Shay Rojansky commented on SPARK-3183: -- +1. As a current workaround for cores, we sp

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114362#comment-14114362 ] Josh Rosen commented on SPARK-3266: --- Thanks for the reproduction! I tried it myself and

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3266: -- Affects Version/s: 1.0.2 > JavaDoubleRDD doesn't contain max() > --- >

[jira] [Resolved] (SPARK-3285) Using values.sum is easier to understand than using values.foldLeft(0)(_ + _)

2014-08-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3285. Resolution: Fixed Fix Version/s: 1.2.0 > Using values.sum is easier to understand than using

[jira] [Resolved] (SPARK-3281) Remove Netty specific code in BlockManager

2014-08-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3281. Resolution: Fixed Fix Version/s: 1.2.0 > Remove Netty specific code in BlockManager > --

[jira] [Assigned] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-3266: - Assignee: Josh Rosen > JavaDoubleRDD doesn't contain max() > ---

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-08-28 Thread Colin B. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin B. updated SPARK-3266: Attachment: spark-repro-3266.tar.gz I have attached a simple java project which reproduces the issue. [^sp

[jira] [Created] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-08-28 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3288: -- Summary: All fields in TaskMetrics should be private and use getters/setters Key: SPARK-3288 URL: https://issues.apache.org/jira/browse/SPARK-3288 Project: Spark

[jira] [Comment Edited] (SPARK-3254) Streaming K-Means

2014-08-28 Thread Jeremy Freeman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114299#comment-14114299 ] Jeremy Freeman edited comment on SPARK-3254 at 8/28/14 8:38 PM:

[jira] [Commented] (SPARK-3254) Streaming K-Means

2014-08-28 Thread Jeremy Freeman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114299#comment-14114299 ] Jeremy Freeman commented on SPARK-3254: --- Here is a (public) google doc explaining a

[jira] [Updated] (SPARK-3287) When ResourceManager High Availability is enabled, ApplicationMaster webUI is not displayed.

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3287: Description: When ResourceManager High Availability is enabled, there will be multiple resource ma

[jira] [Updated] (SPARK-3287) When ResourceManager High Availability is enabled, ApplicationMaster webUI is not displayed.

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3287: Attachment: SPARK-3287.patch In the attached patch, the resource manager list is read using new API

[jira] [Created] (SPARK-3287) When ResourceManager High Availability is enabled, ApplicationMaster webUI is not displayed.

2014-08-28 Thread Benoy Antony (JIRA)
Benoy Antony created SPARK-3287: --- Summary: When ResourceManager High Availability is enabled, ApplicationMaster webUI is not displayed. Key: SPARK-3287 URL: https://issues.apache.org/jira/browse/SPARK-3287

[jira] [Updated] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated SPARK-3286: Attachment: SPARK-3286.patch In the attached patch, the Application Master URL registered by the sp

[jira] [Created] (SPARK-3286) Cannot view ApplicationMaster UI when Yarn’s url scheme is https

2014-08-28 Thread Benoy Antony (JIRA)
Benoy Antony created SPARK-3286: --- Summary: Cannot view ApplicationMaster UI when Yarn’s url scheme is https Key: SPARK-3286 URL: https://issues.apache.org/jira/browse/SPARK-3286 Project: Spark

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114247#comment-14114247 ] Matei Zaharia commented on SPARK-3277: -- Thanks Mridul -- I think Andrew and Patrick h

[jira] [Commented] (SPARK-2475) Check whether #cores > #receivers in local mode

2014-08-28 Thread Chris Fregly (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114204#comment-14114204 ] Chris Fregly commented on SPARK-2475: - another option for the examples, specifically,

[jira] [Commented] (SPARK-3272) Calculate prediction for nodes separately from calculating information gain for splits in decision tree

2014-08-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114166#comment-14114166 ] Joseph K. Bradley commented on SPARK-3272: -- With respect to [SPARK-2207], I think

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-28 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114113#comment-14114113 ] Ted Yu commented on SPARK-1297: --- Here is sample command for building against 0.98 hbase: mv

[jira] [Commented] (SPARK-3280) Made sort-based shuffle the default implementation

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114092#comment-14114092 ] Josh Rosen commented on SPARK-3280: --- Here are some numbers from August 10. If I recall,

[jira] [Updated] (SPARK-3264) Allow users to set executor Spark home in Mesos

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3264: - Fix Version/s: 1.1.0 > Allow users to set executor Spark home in Mesos >

[jira] [Resolved] (SPARK-3264) Allow users to set executor Spark home in Mesos

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-3264. -- Resolution: Fixed > Allow users to set executor Spark home in Mesos > -

[jira] [Updated] (SPARK-2608) Mesos doesn't handle spark.executor.extraJavaOptions correctly (among other things)

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-2608: - Fix Version/s: 1.1.0 > Mesos doesn't handle spark.executor.extraJavaOptions correctly (among other > thi

[jira] [Commented] (SPARK-3280) Made sort-based shuffle the default implementation

2014-08-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114055#comment-14114055 ] Reynold Xin commented on SPARK-3280: [~joshrosen] [~brkyvz] can you guys post the perf

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-3277: -- Fix Version/s: (was: 1.1.0) > LZ4 compression cause the the ExternalSort exception > --

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114039#comment-14114039 ] Andrew Or commented on SPARK-3277: -- This assert was added after 1.0.2. I'm assuming what

[jira] [Resolved] (SPARK-3150) NullPointerException in Spark recovery after simultaneous fall of master and driver

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3150. --- Resolution: Fixed Fix Version/s: 1.0.3 1.1.1 > NullPointerException in Spar

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114026#comment-14114026 ] Mridul Muralidharan commented on SPARK-3277: [~hzw] did you notice this agains

[jira] [Comment Edited] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114022#comment-14114022 ] Mridul Muralidharan edited comment on SPARK-3277 at 8/28/14 5:37 PM: ---

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Attachment: test_lz4_bug.patch Against master, though I noticed similar changes in 1.

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114014#comment-14114014 ] Mridul Muralidharan commented on SPARK-3277: [~matei] Attaching a patch which

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Priority: Blocker (was: Major) > LZ4 compression cause the the ExternalSort exceptio

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Affects Version/s: 1.2.0 1.1.0 > LZ4 compression cause the the

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-28 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113977#comment-14113977 ] Ted Yu commented on SPARK-1297: --- Patch v5 is the aggregate of the 4 commits in the pull requ

[jira] [Updated] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-28 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated SPARK-1297: -- Attachment: spark-1297-v5.txt > Upgrade HBase dependency to 0.98.0 > -- > >

[jira] [Resolved] (SPARK-2855) pyspark test cases crashed for no reason

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2855. --- Resolution: Fixed That issue should be fixed now, so I'm going to mark this JIRA as resolved. Feel

[jira] [Commented] (SPARK-2855) pyspark test cases crashed for no reason

2014-08-28 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113965#comment-14113965 ] Nan Zhu commented on SPARK-2855: no https://github.com/apache/spark/pull/1313 search

[jira] [Commented] (SPARK-2855) pyspark test cases crashed for no reason

2014-08-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113957#comment-14113957 ] Josh Rosen commented on SPARK-2855: --- Do you recall the actual exception? Was it a Py4J

[jira] [Commented] (SPARK-2435) Add shutdown hook to bin/pyspark

2014-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113911#comment-14113911 ] Apache Spark commented on SPARK-2435: - User 'mattf' has created a pull request for thi

[jira] [Commented] (SPARK-2435) Add shutdown hook to bin/pyspark

2014-08-28 Thread Matthew Farrellee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113816#comment-14113816 ] Matthew Farrellee commented on SPARK-2435: -- i couldn't find a PR for this, and it

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread hzw (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113789#comment-14113789 ] hzw commented on SPARK-3277: Sorry,I can not understand it clearly since I'm not familiar with

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread hzw (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hzw updated SPARK-3277: --- Description: I tested the LZ4 compression,and it come up with such problem.(with wordcount) Also I tested the snappy

[jira] [Commented] (SPARK-2855) pyspark test cases crashed for no reason

2014-08-28 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113776#comment-14113776 ] Nan Zhu commented on SPARK-2855: I guess they have fixed this.Jenkins side mistake? >

  1   2   >