[jira] [Commented] (SPARK-2873) OOM happens when group by and join operation with big data

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101915#comment-14101915 ] Apache Spark commented on SPARK-2873: - User 'guowei2' has created a pull request for

[jira] [Created] (SPARK-3119) Re-implement TorrentBroadcast

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3119: -- Summary: Re-implement TorrentBroadcast Key: SPARK-3119 URL: https://issues.apache.org/jira/browse/SPARK-3119 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-3119) Re-implement TorrentBroadcast

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3119: --- Description: TorrentBroadcast is unnecessarily complicated: 1. It tracks a lot of mutable states,

[jira] [Updated] (SPARK-3119) Re-implement TorrentBroadcast

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3119: --- Component/s: Spark Core Re-implement TorrentBroadcast -

[jira] [Commented] (SPARK-3119) Re-implement TorrentBroadcast

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101931#comment-14101931 ] Apache Spark commented on SPARK-3119: - User 'rxin' has created a pull request for this

[jira] [Created] (SPARK-3120) Local Dirs is not useful in yarn-client mode

2014-08-19 Thread hzw (JIRA)
hzw created SPARK-3120: -- Summary: Local Dirs is not useful in yarn-client mode Key: SPARK-3120 URL: https://issues.apache.org/jira/browse/SPARK-3120 Project: Spark Issue Type: Bug Components:

[jira] [Comment Edited] (SPARK-3098) In some cases, operation groupByKey get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102012#comment-14102012 ] Guoqiang Li edited comment on SPARK-3098 at 8/19/14 8:22 AM: -

[jira] [Commented] (SPARK-3098) In some cases, operation groupByKey get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102012#comment-14102012 ] Guoqiang Li commented on SPARK-3098: I found the error id is continuous. Seems there

[jira] [Commented] (SPARK-3098) In some cases, operation groupByKey get a wrong results

2014-08-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102015#comment-14102015 ] Sean Owen commented on SPARK-3098: -- zipWithIndex returns an RDD[(T,Long)]. It does not

[jira] [Commented] (SPARK-3098) In some cases, operation groupByKey get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102024#comment-14102024 ] Guoqiang Li commented on SPARK-3098: the (id, value) pairs are generated there

[jira] [Updated] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3098: --- Summary: In some cases, operation zipWithIndex get a wrong results (was: In some cases, operation

[jira] [Commented] (SPARK-3037) Add ArrayType containing null value support to Parquet.

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102034#comment-14102034 ] Apache Spark commented on SPARK-3037: - User 'ueshin' has created a pull request for

[jira] [Commented] (SPARK-3036) Add MapType containing null value support to Parquet.

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102033#comment-14102033 ] Apache Spark commented on SPARK-3036: - User 'ueshin' has created a pull request for

[jira] [Comment Edited] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102024#comment-14102024 ] Guoqiang Li edited comment on SPARK-3098 at 8/19/14 8:55 AM: -

[jira] [Comment Edited] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102024#comment-14102024 ] Guoqiang Li edited comment on SPARK-3098 at 8/19/14 8:58 AM: -

[jira] [Updated] (SPARK-2964) Fix wrong option (-S, --silent), and improve spark-sql and start-thriftserver to leverage bin/util.sh

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2964: -- Summary: Fix wrong option (-S, --silent), and improve spark-sql and start-thriftserver to

[jira] [Updated] (SPARK-2964) Fix wrong option (-S, --silent), and improve spark-sql and start-thriftserver to leverage bin/util.sh

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2964: -- Description: In spark-sql script, they expect -s option but it's wrong. It's typo for -S

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102044#comment-14102044 ] Sean Owen commented on SPARK-3098: -- It would be helpful if you would explain what you are

[jira] [Comment Edited] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102064#comment-14102064 ] Guoqiang Li edited comment on SPARK-3098 at 8/19/14 9:41 AM: -

[jira] [Created] (SPARK-3121) Wrong implementation of implicit bytesWritableConverter

2014-08-19 Thread Jakub Dubovsky (JIRA)
Jakub Dubovsky created SPARK-3121: - Summary: Wrong implementation of implicit bytesWritableConverter Key: SPARK-3121 URL: https://issues.apache.org/jira/browse/SPARK-3121 Project: Spark

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102101#comment-14102101 ] Guoqiang Li commented on SPARK-3098: Seems to be {{zipWithUniqueId}} also has this

[jira] [Updated] (SPARK-3106) *Race Condition Issue* Fix the order of resources in Connection

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3106: -- Summary: *Race Condition Issue* Fix the order of resources in Connection (was: Suppress

[jira] [Updated] (SPARK-3106) *Race Condition Issue* Fix the order of resources in Connection

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3106: -- Description: Now, when we run Spark application, error message is appear on driver's log. The

[jira] [Updated] (SPARK-3106) *Race Condition Issue* Fix the order of closing resources when Connection is closed

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3106: -- Description: Now, when we run Spark application, error message is appear on driver's log. The

[jira] [Updated] (SPARK-3106) *Race Condition Issue* Fix the order of closing resources when Connection is closed

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3106: -- Summary: *Race Condition Issue* Fix the order of closing resources when Connection is closed

[jira] [Commented] (SPARK-3115) Improve task broadcast latency for small tasks

2014-08-19 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102142#comment-14102142 ] Mridul Muralidharan commented on SPARK-3115: I had a tab open with pretty much

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102145#comment-14102145 ] Sean Owen commented on SPARK-3098: -- Yes I get the same result with Spark 1.0.0 with

[jira] [Updated] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3098: --- Description: {code} val c = sc.parallelize(1 to 7899).flatMap { i = (1 to

[jira] [Updated] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3098: --- Description: The reproduce code: {code} val c = sc.parallelize(1 to 7899).flatMap { i =

[jira] [Updated] (SPARK-3099) Staging Directory is never deleed when we run job with YARN Client Mode

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3099: -- Summary: Staging Directory is never deleed when we run job with YARN Client Mode (was: Add a

[jira] [Updated] (SPARK-3101) Missing volatile annotation in ApplicationMaster

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3101: -- Summary: Missing volatile annotation in ApplicationMaster (was: Flag variable in

[jira] [Updated] (SPARK-3099) Staging Directory is never deleted when we run job with YARN Client Mode

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3099: -- Summary: Staging Directory is never deleted when we run job with YARN Client Mode (was:

[jira] [Updated] (SPARK-3090) Avoid not stopping SparkContext with YARN Client mode

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3090: -- Summary: Avoid not stopping SparkContext with YARN Client mode (was: Add shutdown hook to

[jira] [Updated] (SPARK-3089) Fix meaningless error message in ConnectionManager

2014-08-19 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-3089: -- Summary: Fix meaningless error message in ConnectionManager (was: Make error message in

[jira] [Commented] (SPARK-1782) svd for sparse matrix using ARPACK

2014-08-19 Thread Tarek Elgamal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102179#comment-14102179 ] Tarek Elgamal commented on SPARK-1782: -- I am interested to try this new svd

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102187#comment-14102187 ] Guoqiang Li commented on SPARK-3098: [~srowen] the following code also has this issue.

[jira] [Closed] (SPARK-2753) Is it supposed --archives option in yarn cluster mode to uncompress file?

2014-08-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] José Manuel Abuín Mosquera closed SPARK-2753. - Resolution: Not a Problem Is it supposed --archives option in yarn

[jira] [Commented] (SPARK-3120) Local Dirs is not useful in yarn-client mode

2014-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102199#comment-14102199 ] Thomas Graves commented on SPARK-3120: -- Can you please clarify this? You are trying

[jira] [Resolved] (SPARK-3072) Yarn AM not always properly exiting after unregistering from RM

2014-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-3072. -- Resolution: Fixed Fix Version/s: 1.1.0 Yarn AM not always properly exiting after

[jira] [Commented] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102279#comment-14102279 ] Guoqiang Li commented on SPARK-3098: this issue caused by the code:

[jira] [Comment Edited] (SPARK-3098) In some cases, operation zipWithIndex get a wrong results

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102279#comment-14102279 ] Guoqiang Li edited comment on SPARK-3098 at 8/19/14 3:02 PM: -

[jira] [Commented] (SPARK-3122) hadoop-yarn dependencies cannot be resolved

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102288#comment-14102288 ] Guoqiang Li commented on SPARK-3122: Why add {{spark-yarn_2.10}} dependency?

[jira] [Commented] (SPARK-3122) hadoop-yarn dependencies cannot be resolved

2014-08-19 Thread Ran Levi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102300#comment-14102300 ] Ran Levi commented on SPARK-3122: - It was my understanding that it is required to create a

[jira] [Created] (SPARK-3123) override the setName function to set EdgeRDD's name manually just as VertexRDD does.

2014-08-19 Thread uncleGen (JIRA)
uncleGen created SPARK-3123: --- Summary: override the setName function to set EdgeRDD's name manually just as VertexRDD does. Key: SPARK-3123 URL: https://issues.apache.org/jira/browse/SPARK-3123 Project:

[jira] [Commented] (SPARK-3123) override the setName function to set EdgeRDD's name manually just as VertexRDD does.

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102308#comment-14102308 ] Apache Spark commented on SPARK-3123: - User 'uncleGen' has created a pull request for

[jira] [Commented] (SPARK-3118) add SHOW TBLPROPERTIES tblname; and SHOW COLUMNS (FROM|IN) table_name [(FROM|IN) db_name] support

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102317#comment-14102317 ] Apache Spark commented on SPARK-3118: - User 'u0jing' has created a pull request for

[jira] [Commented] (SPARK-3122) hadoop-yarn dependencies cannot be resolved

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102318#comment-14102318 ] Guoqiang Li commented on SPARK-3122: It is not necessary. Only need to these:

[jira] [Commented] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102330#comment-14102330 ] Guoqiang Li commented on SPARK-3124: What's your command? Jar version conflict in

[jira] [Comment Edited] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102330#comment-14102330 ] Guoqiang Li edited comment on SPARK-3124 at 8/19/14 3:42 PM: -

[jira] [Commented] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102334#comment-14102334 ] Apache Spark commented on SPARK-3124: - User 'chenghao-intel' has created a pull

[jira] [Commented] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Cheng Hao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102338#comment-14102338 ] Cheng Hao commented on SPARK-3124: -- Can you try bin/spark-sql after make distribution?

[jira] [Created] (SPARK-3125) hive thriftserver test suite failure

2014-08-19 Thread wangfei (JIRA)
wangfei created SPARK-3125: -- Summary: hive thriftserver test suite failure Key: SPARK-3125 URL: https://issues.apache.org/jira/browse/SPARK-3125 Project: Spark Issue Type: Bug Components:

[jira] [Created] (SPARK-3126) HiveThriftServer2Suite hangs

2014-08-19 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-3126: - Summary: HiveThriftServer2Suite hangs Key: SPARK-3126 URL: https://issues.apache.org/jira/browse/SPARK-3126 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102344#comment-14102344 ] Guoqiang Li commented on SPARK-3124: We should modify the file

[jira] [Created] (SPARK-3127) Modifying Spark SQL related scripts should trigger Spark SQL test suites

2014-08-19 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-3127: - Summary: Modifying Spark SQL related scripts should trigger Spark SQL test suites Key: SPARK-3127 URL: https://issues.apache.org/jira/browse/SPARK-3127 Project: Spark

[jira] [Commented] (SPARK-3120) Local Dirs is not useful in yarn-client mode

2014-08-19 Thread hzw (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102350#comment-14102350 ] hzw commented on SPARK-3120: Do you mean that : If I want to change the local-dirs in Yarn

[jira] [Commented] (SPARK-3124) Jar version conflict in the assembly package

2014-08-19 Thread Cheng Hao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102352#comment-14102352 ] Cheng Hao commented on SPARK-3124: -- Yes, actually I did in the PR. Jar version conflict

[jira] [Commented] (SPARK-2929) Rewrite HiveThriftServer2Suite and CliSuite

2014-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102370#comment-14102370 ] Cheng Lian commented on SPARK-2929: --- Opened SPARK-3126 SPARK-3127 to track failure of

[jira] [Commented] (SPARK-3127) Modifying Spark SQL related scripts should trigger Spark SQL test suites

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102373#comment-14102373 ] Apache Spark commented on SPARK-3127: - User 'liancheng' has created a pull request for

[jira] [Commented] (SPARK-3125) hive thriftserver test suite failure

2014-08-19 Thread wangfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102424#comment-14102424 ] wangfei commented on SPARK-3125: for clisuite i print the error info, as follows:

[jira] [Commented] (SPARK-1782) svd for sparse matrix using ARPACK

2014-08-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102427#comment-14102427 ] Xiangrui Meng commented on SPARK-1782: -- The plan is to release v1.1 by the end of the

[jira] [Created] (SPARK-3128) Use streaming test suite for StreamingLR

2014-08-19 Thread Jeremy Freeman (JIRA)
Jeremy Freeman created SPARK-3128: - Summary: Use streaming test suite for StreamingLR Key: SPARK-3128 URL: https://issues.apache.org/jira/browse/SPARK-3128 Project: Spark Issue Type:

[jira] [Updated] (SPARK-3129) Prevent data loss in Spark Streaming

2014-08-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-3129: Issue Type: New Feature (was: Bug) Prevent data loss in Spark Streaming

[jira] [Updated] (SPARK-3129) Prevent data loss in Spark Streaming

2014-08-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-3129: Attachment: StreamingPreventDataLoss.pdf Prevent data loss in Spark Streaming

[jira] [Commented] (SPARK-3122) hadoop-yarn dependencies cannot be resolved

2014-08-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102472#comment-14102472 ] Sean Owen commented on SPARK-3122: -- [~gq] You do not need to depend on hadoop-client for

[jira] [Created] (SPARK-3130) Should not allow negative values in naive Bayes

2014-08-19 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-3130: Summary: Should not allow negative values in naive Bayes Key: SPARK-3130 URL: https://issues.apache.org/jira/browse/SPARK-3130 Project: Spark Issue Type:

[jira] [Updated] (SPARK-3110) Add a ha mode in YARN mode to keep executors in between restarts

2014-08-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-3110: Issue Type: Sub-task (was: Bug) Parent: SPARK-3129 Add a ha mode in YARN mode to

[jira] [Commented] (SPARK-3128) Use streaming test suite for StreamingLR

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102481#comment-14102481 ] Apache Spark commented on SPARK-3128: - User 'freeman-lab' has created a pull request

[jira] [Resolved] (SPARK-3089) Fix meaningless error message in ConnectionManager

2014-08-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3089. --- Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Kousuke Saruta Fix meaningless

[jira] [Commented] (SPARK-3130) Should not allow negative values in naive Bayes

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102499#comment-14102499 ] Apache Spark commented on SPARK-3130: - User 'mengxr' has created a pull request for

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

2014-08-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102501#comment-14102501 ] Hari Shreedharan commented on SPARK-3129: - This doc is an early list of fixes. I

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

2014-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102504#comment-14102504 ] Thomas Graves commented on SPARK-3129: -- A couple of random thoughts on this for yarn.

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

2014-08-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102579#comment-14102579 ] Hari Shreedharan commented on SPARK-3129: - The way the driver finds the executors

[jira] [Updated] (SPARK-3131) Allow user to set parquet compression codec for writing ParquetFile in SQLContext

2014-08-19 Thread Teng Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Qiu updated SPARK-3131: Summary: Allow user to set parquet compression codec for writing ParquetFile in SQLContext (was: Allow

[jira] [Updated] (SPARK-3131) Allow user to set parquet compression codec for writing ParquetFile in SQLContext

2014-08-19 Thread Teng Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Qiu updated SPARK-3131: Description: There are 4 different compression codec available for ParquetOutputFormat in Spark SQL it

[jira] [Commented] (SPARK-3131) Allow user to set parquet compression codec for writing ParquetFile in SQLContext

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102724#comment-14102724 ] Apache Spark commented on SPARK-3131: - User 'chutium' has created a pull request for

[jira] [Commented] (SPARK-3117) Avoid serialization for TorrentBroadcast blocks

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102758#comment-14102758 ] Reynold Xin commented on SPARK-3117: This is going to be fixed by

[jira] [Created] (SPARK-3132) Avoid serialization for Array[Byte] in TorrentBroadcast

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3132: -- Summary: Avoid serialization for Array[Byte] in TorrentBroadcast Key: SPARK-3132 URL: https://issues.apache.org/jira/browse/SPARK-3132 Project: Spark Issue

[jira] [Created] (SPARK-3133) Piggyback get location RPC call to fetch small blocks

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3133: -- Summary: Piggyback get location RPC call to fetch small blocks Key: SPARK-3133 URL: https://issues.apache.org/jira/browse/SPARK-3133 Project: Spark Issue Type:

[jira] [Created] (SPARK-3134) Update block locations asynchronously in TorrentBroadcast

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3134: -- Summary: Update block locations asynchronously in TorrentBroadcast Key: SPARK-3134 URL: https://issues.apache.org/jira/browse/SPARK-3134 Project: Spark Issue

[jira] [Created] (SPARK-3135) Avoid memory copy in TorrentBroadcast serialization

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3135: -- Summary: Avoid memory copy in TorrentBroadcast serialization Key: SPARK-3135 URL: https://issues.apache.org/jira/browse/SPARK-3135 Project: Spark Issue Type:

[jira] [Created] (SPARK-3136) create java-friendly methods in RandomRDDs

2014-08-19 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-3136: Summary: create java-friendly methods in RandomRDDs Key: SPARK-3136 URL: https://issues.apache.org/jira/browse/SPARK-3136 Project: Spark Issue Type:

[jira] [Updated] (SPARK-3135) Avoid memory copy in TorrentBroadcast serialization

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3135: --- Description: TorrentBroadcast.blockifyObject uses a ByteArrayOutputStream to serialize broadcast

[jira] [Updated] (SPARK-3135) Avoid memory copy in TorrentBroadcast serialization

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3135: --- Labels: starter (was: ) Avoid memory copy in TorrentBroadcast serialization

[jira] [Updated] (SPARK-3133) Piggyback get location RPC call to fetch small blocks

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3133: --- Description: We should add a new API to the BlockManagerMasterActor to get location or the data

[jira] [Resolved] (SPARK-3128) Use streaming test suite for StreamingLR

2014-08-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-3128. -- Resolution: Fixed Fix Version/s: 1.2.0 1.1.0 Use streaming test

[jira] [Updated] (SPARK-3133) Piggyback get location RPC call to fetch small blocks

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3133: --- Description: We should add a new API to the BlockManagerMasterActor to get location or the data

[jira] [Created] (SPARK-3137) Use finer grained locking in TorrentBroadcast.readObject

2014-08-19 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-3137: -- Summary: Use finer grained locking in TorrentBroadcast.readObject Key: SPARK-3137 URL: https://issues.apache.org/jira/browse/SPARK-3137 Project: Spark Issue

[jira] [Updated] (SPARK-3137) Use finer grained locking in TorrentBroadcast.readObject

2014-08-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-3137: --- Component/s: Spark Core Target Version/s: 1.2.0 Use finer grained locking in

[jira] [Commented] (SPARK-3136) create java-friendly methods in RandomRDDs

2014-08-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102791#comment-14102791 ] Apache Spark commented on SPARK-3136: - User 'mengxr' has created a pull request for

[jira] [Resolved] (SPARK-2333) spark_ec2 script should allow option for existing security group

2014-08-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2333. --- Resolution: Fixed spark_ec2 script should allow option for existing security group

[jira] [Updated] (SPARK-2333) spark_ec2 script should allow option for existing security group

2014-08-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2333: -- Issue Type: Improvement (was: Bug) spark_ec2 script should allow option for existing security group

[jira] [Updated] (SPARK-2839) Documentation for statistical functions

2014-08-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2839: - Assignee: Burak Yavuz Documentation for statistical functions

[jira] [Updated] (SPARK-3112) Documentation for Streaming Logistic Regression Streaming

2014-08-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3112: - Assignee: Jeremy Freeman Documentation for Streaming Logistic Regression Streaming

[jira] [Resolved] (SPARK-2790) PySpark zip() doesn't work properly if RDDs have different serializers

2014-08-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2790. --- Resolution: Fixed Fix Version/s: 1.1.0 PySpark zip() doesn't work properly if RDDs have

[jira] [Created] (SPARK-3138) sqlContext.parquetFile should be able to take a single file as parameter

2014-08-19 Thread Teng Qiu (JIRA)
Teng Qiu created SPARK-3138: --- Summary: sqlContext.parquetFile should be able to take a single file as parameter Key: SPARK-3138 URL: https://issues.apache.org/jira/browse/SPARK-3138 Project: Spark

[jira] [Created] (SPARK-3139) Akka timeouts from ContextCleaner when cleaning shuffles

2014-08-19 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-3139: - Summary: Akka timeouts from ContextCleaner when cleaning shuffles Key: SPARK-3139 URL: https://issues.apache.org/jira/browse/SPARK-3139 Project: Spark Issue Type:

[jira] [Commented] (SPARK-3138) sqlContext.parquetFile should be able to take a single file as parameter

2014-08-19 Thread Teng Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102961#comment-14102961 ] Teng Qiu commented on SPARK-3138: - be careful if someone is working on SPARK-2551, make

[jira] [Commented] (SPARK-3139) Akka timeouts from ContextCleaner when cleaning shuffles

2014-08-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102960#comment-14102960 ] Josh Rosen commented on SPARK-3139: --- I used pssh + grep to search through the

[jira] [Created] (SPARK-3140) PySpark start-up throws confusing exception

2014-08-19 Thread Andrew Or (JIRA)
Andrew Or created SPARK-3140: Summary: PySpark start-up throws confusing exception Key: SPARK-3140 URL: https://issues.apache.org/jira/browse/SPARK-3140 Project: Spark Issue Type: Bug

  1   2   >