[jira] [Commented] (SPARK-3481) HiveComparisonTest throws exception of "org.apache.hadoop.hive.ql.metadata.HiveException: Database does not exist: default"

2014-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132567#comment-14132567 ] Apache Spark commented on SPARK-3481: - User 'liancheng' has created a pull request for

[jira] [Updated] (SPARK-2883) Spark Support for ORCFile format

2014-09-12 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2883: Target Version/s: 1.2.0 > Spark Support for ORCFile format > ---

[jira] [Updated] (SPARK-2883) Spark Support for ORCFile format

2014-09-12 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2883: Priority: Blocker (was: Major) > Spark Support for ORCFile format > ---

[jira] [Resolved] (SPARK-3455) **HotFix** Unit test failed due to can not resolve the attribute references

2014-09-12 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-3455. - Resolution: Fixed > **HotFix** Unit test failed due to can not resolve the attribute refer

[jira] [Updated] (SPARK-3500) coalesce() and repartition() of SchemaRDD is broken

2014-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3500: -- Description: {code} >>> sqlCtx.jsonRDD(sc.parallelize(['{"foo":"bar"}', >>> '{"foo":"baz"}'])).coalesce

[jira] [Updated] (SPARK-3500) coalesce() and repartition() of SchemaRDD is broken

2014-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3500: -- Summary: coalesce() and repartition() of SchemaRDD is broken (was: SchemaRDD from jsonRDD() has not coa

[jira] [Resolved] (SPARK-3469) All TaskCompletionListeners should be called even if some of them fail

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3469. Resolution: Fixed Fix Version/s: 1.2.0 Target Version/s: 1.2.0 (was: 1.1.1, 1.2.0)

[jira] [Commented] (SPARK-3517) mapPartitions is not correct clearing up the closure

2014-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132489#comment-14132489 ] Apache Spark commented on SPARK-3517: - User 'witgo' has created a pull request for thi

[jira] [Resolved] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3500. --- Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 Issue resolved by pull request

[jira] [Updated] (SPARK-2883) Spark Support for ORCFile format

2014-09-12 Thread Fi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fi updated SPARK-2883: -- Attachment: 2014-09-12 07.07.19 pm jobtracker.png 2014-09-12 07.05.24 pm Spark UI.png > Spark Support fo

[jira] [Commented] (SPARK-2883) Spark Support for ORCFile format

2014-09-12 Thread Fi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132472#comment-14132472 ] Fi commented on SPARK-2883: --- I was able to run a simple query and access my ORC hive table throu

[jira] [Updated] (SPARK-3517) mapPartitions is not correct clearing up the closure

2014-09-12 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3517: --- Summary: mapPartitions is not correct clearing up the closure (was: mapPartitions is not correct clea

[jira] [Updated] (SPARK-3517) mapPartitions is not correct clearing closure

2014-09-12 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3517: --- Description: {code} for (iter <- 1 to totalIter) { logInfo("Start Gibbs sampling (Iteration %d/

[jira] [Updated] (SPARK-3517) mapPartitions is not correct clearing closure

2014-09-12 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3517: --- Description: {code} for (iter <- 1 to totalIter) { logInfo("Start Gibbs sampling (Iteration %d/

[jira] [Created] (SPARK-3517) mapPartitions is not correct clearing closure

2014-09-12 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-3517: -- Summary: mapPartitions is not correct clearing closure Key: SPARK-3517 URL: https://issues.apache.org/jira/browse/SPARK-3517 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-3094) Support run pyspark in PyPy

2014-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3094. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2144 [https://github.com/

[jira] [Resolved] (SPARK-3456) YarnAllocator can lose container requests to RM

2014-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-3456. -- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Thomas Graves > YarnAllocator c

[jira] [Created] (SPARK-3516) DecisionTree Python support for params maxInstancesPerNode, maxInfoGain

2014-09-12 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-3516: Summary: DecisionTree Python support for params maxInstancesPerNode, maxInfoGain Key: SPARK-3516 URL: https://issues.apache.org/jira/browse/SPARK-3516 Project

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132420#comment-14132420 ] Josh Rosen commented on SPARK-3500: --- This feels like a bug, not a missing feature, since

[jira] [Commented] (SPARK-3515) ParquetMetastoreSuite fails when executed together with other suites under Maven

2014-09-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132406#comment-14132406 ] Cheng Lian commented on SPARK-3515: --- The bug SPARK-3481 fixed actually covered up the bu

[jira] [Commented] (SPARK-3515) ParquetMetastoreSuite fails when executed together with other suites under Maven

2014-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132378#comment-14132378 ] Apache Spark commented on SPARK-3515: - User 'liancheng' has created a pull request for

[jira] [Created] (SPARK-3515) ParquetMetastoreSuite fails when executed together with other suites under Maven

2014-09-12 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-3515: - Summary: ParquetMetastoreSuite fails when executed together with other suites under Maven Key: SPARK-3515 URL: https://issues.apache.org/jira/browse/SPARK-3515 Project: Spa

[jira] [Updated] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

2014-09-12 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-1021: Assignee: Erik Erlandson (was: Mark Hamstra) > sortByKey() launches a cluster job when it shouldn't

[jira] [Commented] (SPARK-1449) Please delete old releases from mirroring system

2014-09-12 Thread Sebb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132323#comment-14132323 ] Sebb commented on SPARK-1449: - Is no-one able to deal with this please? > Please delete old r

[jira] [Updated] (SPARK-1449) Please delete old releases from mirroring system

2014-09-12 Thread Sebb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebb updated SPARK-1449: Affects Version/s: 1.0.1 0.9.2 1.0.0 1.0.2 > Pl

[jira] [Updated] (SPARK-1449) Please delete old releases from mirroring system

2014-09-12 Thread Sebb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebb updated SPARK-1449: Affects Version/s: (was: 0.9.0) > Please delete old releases from mirroring system > ---

[jira] [Updated] (SPARK-1449) Please delete old releases from mirroring system

2014-09-12 Thread Sebb (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebb updated SPARK-1449: Affects Version/s: (was: 0.8.0) > Please delete old releases from mirroring system > ---

[jira] [Updated] (SPARK-3464) Graceful decommission of executors

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3464: - Target Version/s: 1.2.0 Affects Version/s: 1.1.0 Assignee: Andrew Or > Graceful decommis

[jira] [Resolved] (SPARK-3217) Shaded Guava jar doesn't play well with Maven build when SPARK_PREPEND_CLASSES is set

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3217. Resolution: Fixed Fix Version/s: 1.2.0 Fixed by https://github.com/apache/spark/pull/

[jira] [Closed] (SPARK-1131) Better document the --args option for yarn-standalone mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1131. Resolution: Fixed --args is now deprecated. We use --arg instead. > Better document the --args option for y

[jira] [Closed] (SPARK-1909) "--jars" is not supported in standalone cluster mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1909. Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Andrew Or Also fixed in https://github.com/

[jira] [Closed] (SPARK-1908) Support local app jar in standalone cluster mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1908. Resolution: Won't Fix Assignee: Andrew Or As described in the comment of https://github.com/apache/spa

[jira] [Resolved] (SPARK-3014) Log a more informative messages in a couple failure scenarios

2014-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-3014. -- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Sandy Ryza > Log a more informa

[jira] [Closed] (SPARK-1906) spark-submit doesn't send master URL to Driver in standalone cluster mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1906. Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Andrew Or This is fixed in https://github.c

[jira] [Commented] (SPARK-3107) Don't pass null jar to executor in yarn-client mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132164#comment-14132164 ] Andrew Or commented on SPARK-3107: -- I see. Yes, setting it to an empty value is semantica

[jira] [Updated] (SPARK-3187) Refactor and cleanup Yarn allocator code

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3187: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-3492 > Refactor and cleanup Yarn allocator

[jira] [Commented] (SPARK-3492) Clean up Yarn integration code

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132147#comment-14132147 ] Andrew Or commented on SPARK-3492: -- Thanks, I've added it to the list. > Clean up Yarn i

[jira] [Closed] (SPARK-3465) Task metrics are not aggregated correctly in local mode

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3465. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 > Task metrics are not aggregated cor

[jira] [Closed] (SPARK-3498) Block always replicated to the same node

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3498. Resolution: Duplicate Assignee: Tathagata Das This is the cause of SPARK-3495, and the fix for both is

[jira] [Updated] (SPARK-3513) Provide a utility for running a function once on each executor

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3513: --- Issue Type: Improvement (was: Bug) > Provide a utility for running a function once on each ex

[jira] [Updated] (SPARK-3514) Provide a utility function for returning the hosts (and number) of live executors

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3514: --- Issue Type: Improvement (was: Bug) > Provide a utility function for returning the hosts (and

[jira] [Created] (SPARK-3514) Provide a utility function for returning the hosts (and number) of live executors

2014-09-12 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3514: -- Summary: Provide a utility function for returning the hosts (and number) of live executors Key: SPARK-3514 URL: https://issues.apache.org/jira/browse/SPARK-3514 P

[jira] [Created] (SPARK-3513) Provide a utility for running a function once on each executor

2014-09-12 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3513: -- Summary: Provide a utility for running a function once on each executor Key: SPARK-3513 URL: https://issues.apache.org/jira/browse/SPARK-3513 Project: Spark

[jira] [Updated] (SPARK-3513) Provide a utility for running a function once on each executor

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3513: --- Target Version/s: 1.2.0 > Provide a utility for running a function once on each executor > ---

[jira] [Updated] (SPARK-3513) Provide a utility for running a function once on each executor

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3513: --- Fix Version/s: (was: 1.2.0) > Provide a utility for running a function once on each execut

[jira] [Resolved] (SPARK-2699) Improve compatibility with parquet file/table

2014-09-12 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2699. - Resolution: Duplicate Closing as already fixed duplicate :) > Improve compatibility with

[jira] [Updated] (SPARK-2992) The transforms formerly known as non-lazy

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2992: --- Affects Version/s: 1.1.0 > The transforms formerly known as non-lazy > ---

[jira] [Resolved] (SPARK-3427) Avoid active vertex tracking in static PageRank

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3427. Resolution: Fixed Fix Version/s: 1.2.0 > Avoid active vertex tracking in static PageRank > --

[jira] [Updated] (SPARK-2992) The transforms formerly known as non-lazy

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2992: --- Priority: Major (was: Minor) > The transforms formerly known as non-lazy > --

[jira] [Updated] (SPARK-1579) PySpark should distinguish expected IOExceptions from unexpected ones in the worker

2014-09-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-1579: -- Fix Version/s: (was: 1.1.0) 1.0.0 > PySpark should distinguish expected IOExcepti

[jira] [Updated] (SPARK-3339) Support for skipping json lines that fail to parse

2014-09-12 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-3339: Priority: Critical (was: Major) > Support for skipping json lines that fail to parse >

[jira] [Commented] (SPARK-2083) Allow local task to retry after failure.

2014-09-12 Thread Radim Kolar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132005#comment-14132005 ] Radim Kolar commented on SPARK-2083: i took look at patch. why to add new settings to

[jira] [Commented] (SPARK-3456) YarnAllocator can lose container requests to RM

2014-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131993#comment-14131993 ] Apache Spark commented on SPARK-3456: - User 'tgravescs' has created a pull request for

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131933#comment-14131933 ] Nicholas Chammas commented on SPARK-3500: - [~davies] - PySpark doesn't seem to sup

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a firewall,

[jira] [Updated] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-3500: -- Description: {code} >>> sqlCtx.jsonRDD(sc.parallelize(['{"foo":"bar"}', >>> '{"foo":"baz"}'])).coalesce

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131869#comment-14131869 ] Davies Liu commented on SPARK-3500: --- repartition() and distinct(N) are also missing too.

[jira] [Updated] (SPARK-3490) Alleviate port collisions during tests

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3490: - Fix Version/s: (was: 1.1.1) > Alleviate port collisions during tests > ---

[jira] [Commented] (SPARK-3490) Alleviate port collisions during tests

2014-09-12 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131844#comment-14131844 ] Andrew Or commented on SPARK-3490: -- This still needs to be back ported into branch-1.1 >

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131823#comment-14131823 ] Reynold Xin commented on SPARK-2926: Do you mind creating a separate branch that's bas

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131757#comment-14131757 ] Nicholas Chammas commented on SPARK-3500: - Btw, this seems like the same type of p

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a firewall,

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131752#comment-14131752 ] Nicholas Chammas commented on SPARK-3500: - Hmm, you _could_ perhaps consider this

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a firewall,

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a firewall,

[jira] [Updated] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjia Wang updated SPARK-3512: Description: I believe this would be a common scenario that the yarn cluster runs behind a firewall,

[jira] [Created] (SPARK-3512) yarn-client through socks proxy

2014-09-12 Thread Yongjia Wang (JIRA)
Yongjia Wang created SPARK-3512: --- Summary: yarn-client through socks proxy Key: SPARK-3512 URL: https://issues.apache.org/jira/browse/SPARK-3512 Project: Spark Issue Type: Wish Compon

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131714#comment-14131714 ] Davies Liu commented on SPARK-3500: --- I think it's a bug, there is a workaround for it:

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131706#comment-14131706 ] Patrick Wendell commented on SPARK-3500: If it's just a missing feature we tend to

[jira] [Commented] (SPARK-3500) SchemaRDD from jsonRDD() has not coalesce() method

2014-09-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131688#comment-14131688 ] Nicholas Chammas commented on SPARK-3500: - [~davies] - Shouldn't the target versio

[jira] [Created] (SPARK-3511) Create a RELEASE-NOTES.txt file in the repo

2014-09-12 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3511: -- Summary: Create a RELEASE-NOTES.txt file in the repo Key: SPARK-3511 URL: https://issues.apache.org/jira/browse/SPARK-3511 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3507) Create RegressionLearner trait and make some currect code implement it

2014-09-12 Thread Egor Pakhomov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131658#comment-14131658 ] Egor Pakhomov commented on SPARK-3507: -- https://github.com/apache/spark/pull/2371 >

[jira] [Issue Comment Deleted] (SPARK-3507) Create RegressionLearner trait and make some currect code implement it

2014-09-12 Thread Egor Pakhomov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Egor Pakhomov updated SPARK-3507: - Comment: was deleted (was: https://github.com/apache/spark/pull/2371) > Create RegressionLearner

[jira] [Commented] (SPARK-3507) Create RegressionLearner trait and make some currect code implement it

2014-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131650#comment-14131650 ] Apache Spark commented on SPARK-3507: - User 'epahomov' has created a pull request for

[jira] [Comment Edited] (SPARK-3499) Create Spark-based distcp utility

2014-09-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131054#comment-14131054 ] Nicholas Chammas edited comment on SPARK-3499 at 9/12/14 2:27 PM: --

[jira] [Created] (SPARK-3510) Create method for calculating error between expected result and actual

2014-09-12 Thread Egor Pakhomov (JIRA)
Egor Pakhomov created SPARK-3510: Summary: Create method for calculating error between expected result and actual Key: SPARK-3510 URL: https://issues.apache.org/jira/browse/SPARK-3510 Project: Spark

[jira] [Commented] (SPARK-3509) Method for generating random LabeledPoints for testing

2014-09-12 Thread Egor Pakhomov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131554#comment-14131554 ] Egor Pakhomov commented on SPARK-3509: -- So far I have bad code for my usages. need go

[jira] [Created] (SPARK-3509) Method for generating random LabeledPoints for testing

2014-09-12 Thread Egor Pakhomov (JIRA)
Egor Pakhomov created SPARK-3509: Summary: Method for generating random LabeledPoints for testing Key: SPARK-3509 URL: https://issues.apache.org/jira/browse/SPARK-3509 Project: Spark Issue Ty

[jira] [Created] (SPARK-3508) annotate the Spark configs to indicate which ones are meant for the end user

2014-09-12 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-3508: Summary: annotate the Spark configs to indicate which ones are meant for the end user Key: SPARK-3508 URL: https://issues.apache.org/jira/browse/SPARK-3508 Project: S

[jira] [Resolved] (SPARK-2558) Mention --queue argument in YARN documentation

2014-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-2558. -- Resolution: Fixed Fix Version/s: 1.2.0 Target Version/s: 1.2.0 (was: 1.1.0, 1.0

[jira] [Created] (SPARK-3507) Create RegressionLearner trait and make some currect code implement it

2014-09-12 Thread Egor Pakhomov (JIRA)
Egor Pakhomov created SPARK-3507: Summary: Create RegressionLearner trait and make some currect code implement it Key: SPARK-3507 URL: https://issues.apache.org/jira/browse/SPARK-3507 Project: Spark

[jira] [Commented] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-09-12 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131451#comment-14131451 ] Prashant Sharma commented on SPARK-2182: Found this SO link useful, http://stacko

[jira] [Created] (SPARK-3506) 1.1.0-SNAPSHOT in docs for 1.1.0 under docs/latest

2014-09-12 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-3506: -- Summary: 1.1.0-SNAPSHOT in docs for 1.1.0 under docs/latest Key: SPARK-3506 URL: https://issues.apache.org/jira/browse/SPARK-3506 Project: Spark Issue Ty

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-09-12 Thread Ryan D Braley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131333#comment-14131333 ] Ryan D Braley commented on SPARK-2593: -- This would be quite useful. It is hard to use

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

2014-09-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131314#comment-14131314 ] Saisai Shao commented on SPARK-2926: Hi Reynold, thanks a lot for your watching this,

[jira] [Commented] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131299#comment-14131299 ] Xiangrui Meng commented on SPARK-1405: -- [~xusen] and [~gq] Thanks for working on LDA!

[jira] [Updated] (SPARK-2838) performance tests for feature transformations

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2838: - Assignee: (was: Xiangrui Meng) > performance tests for feature transformations > -

[jira] [Updated] (SPARK-3436) [MLlib]Streaming SVM

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3436: - Assignee: Liquan Pei > [MLlib]Streaming SVM > - > > Key: SPAR

[jira] [Updated] (SPARK-3249) Fix links in ScalaDoc that cause warning messages in `sbt/sbt unidoc`

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3249: - Target Version/s: 1.2.0 (was: 1.1.0) > Fix links in ScalaDoc that cause warning messages in `sbt/

[jira] [Updated] (SPARK-2838) performance tests for feature transformations

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2838: - Target Version/s: 1.2.0 (was: 1.1.0) > performance tests for feature transformations > --

[jira] [Resolved] (SPARK-2830) MLlib v1.1 documentation

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2830. -- Resolution: Fixed Fix Version/s: 1.1.0 > MLlib v1.1 documentation > -

[jira] [Resolved] (SPARK-3160) Simplify DecisionTree data structure for training

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-3160. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2341 [https://githu

[jira] [Updated] (SPARK-3494) DecisionTree overflow error in calculating maxMemoryUsage

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3494: - Fix Version/s: 1.2.0 > DecisionTree overflow error in calculating maxMemoryUsage > ---

[jira] [Resolved] (SPARK-3494) DecisionTree overflow error in calculating maxMemoryUsage

2014-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-3494. -- Resolution: Fixed Assignee: Joseph K. Bradley https://github.com/apache/spark/pull/2341 >

[jira] [Updated] (SPARK-3393) Align the log4j configuration for Spark & SparkSQLCLI

2014-09-12 Thread Cheng Hao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated SPARK-3393: - Summary: Align the log4j configuration for Spark & SparkSQLCLI (was: Add configuration templates for HQL

[jira] [Commented] (SPARK-3084) Collect broadcasted tables in parallel in joins

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131258#comment-14131258 ] Reynold Xin commented on SPARK-3084: Note that the current fix actually launches jobs

[jira] [Updated] (SPARK-2992) The transforms formerly known as non-lazy

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2992: --- Target Version/s: 1.2.0 (was: 1.1.1) > The transforms formerly known as non-lazy > --

[jira] [Commented] (SPARK-3499) Create Spark-based distcp utility

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131255#comment-14131255 ] Reynold Xin commented on SPARK-3499: Would be pretty cool to have actually. I don't kn

[jira] [Commented] (SPARK-3498) Block always replicated to the same node

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131256#comment-14131256 ] Reynold Xin commented on SPARK-3498: cc [~tdas] > Block always replicated to the same

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

2014-09-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131248#comment-14131248 ] Reynold Xin commented on SPARK-2926: Do you have a branch that I can test with? I'm ru

  1   2   >