[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179660#comment-14179660 ] Apache Spark commented on SPARK-3426: - User 'JoshRosen' has created a pull request for

[jira] [Created] (SPARK-4045) BinaryArithmetic cannot implicitly cast StringType to DoubleType

2014-10-22 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4045: - Summary: BinaryArithmetic cannot implicitly cast StringType to DoubleType Key: SPARK-4045 URL: https://issues.apache.org/jira/browse/SPARK-4045 Project: Spark

[jira] [Commented] (SPARK-4045) BinaryArithmetic cannot implicitly cast StringType to DoubleType

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179712#comment-14179712 ] Apache Spark commented on SPARK-4045: - User 'sarutak' has created a pull request for t

[jira] [Updated] (SPARK-4045) BinaryArithmetic should not implicitly cast StringType to DoubleType

2014-10-22 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-4045: -- Summary: BinaryArithmetic should not implicitly cast StringType to DoubleType (was: BinaryArith

[jira] [Closed] (SPARK-3939) NPE caused by SessionState.out not set in thriftserver2

2014-10-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian closed SPARK-3939. - Resolution: Duplicate > NPE caused by SessionState.out not set in thriftserver2 >

[jira] [Commented] (SPARK-3939) NPE caused by SessionState.out not set in thriftserver2

2014-10-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179749#comment-14179749 ] Cheng Lian commented on SPARK-3939: --- Ah, actually it's SPARK-4037 who duplicates this ti

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

2014-10-22 Thread Ye Xianjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179753#comment-14179753 ] Ye Xianjin commented on SPARK-4002: --- Hi, [~rdub] what's your mac os x's hostname ? Mine

[jira] [Created] (SPARK-4046) Incorrect examples on site

2014-10-22 Thread Ian Babrou (JIRA)
Ian Babrou created SPARK-4046: - Summary: Incorrect examples on site Key: SPARK-4046 URL: https://issues.apache.org/jira/browse/SPARK-4046 Project: Spark Issue Type: Bug Components: Docu

[jira] [Updated] (SPARK-4046) Incorrect Java example on site

2014-10-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4046: - Priority: Minor (was: Critical) Affects Version/s: 1.1.0 Summary: Incorrect Jav

[jira] [Closed] (SPARK-4045) BinaryArithmetic should not implicitly cast StringType to DoubleType

2014-10-22 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta closed SPARK-4045. - Resolution: Won't Fix > BinaryArithmetic should not implicitly cast StringType to DoubleType > ---

[jira] [Commented] (SPARK-3815) LPAD function does not work in where predicate

2014-10-22 Thread Venkata Ramana G (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179813#comment-14179813 ] Venkata Ramana G commented on SPARK-3815: - Still the issue is not re-producible, b

[jira] [Commented] (SPARK-4040) calling count() on RDD's emitted from a DStream blocks forEachRDD progress.

2014-10-22 Thread RJ Nowling (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179882#comment-14179882 ] RJ Nowling commented on SPARK-4040: --- I don't think you can access a RDD from with an ope

[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

2014-10-22 Thread RJ Nowling (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179890#comment-14179890 ] RJ Nowling commented on SPARK-2429: --- A 6x performance improvement is great improvement!

[jira] [Commented] (SPARK-4040) calling count() on RDD's emitted from a DStream blocks forEachRDD progress.

2014-10-22 Thread jay vyas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179898#comment-14179898 ] jay vyas commented on SPARK-4040: - Makes sense. Is it possible that RDD's themselves , wh

[jira] [Updated] (SPARK-4042) append columns ids and names before broadcast

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-4042: Target Version/s: 1.1.1, 1.2.0 (was: 1.1.1) > append columns ids and names before broadcast > -

[jira] [Commented] (SPARK-4042) append columns ids and names before broadcast

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179939#comment-14179939 ] Yin Huai commented on SPARK-4042: - Can you also add some test results? Like the amount of

[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

2014-10-22 Thread Yu Ishikawa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179974#comment-14179974 ] Yu Ishikawa commented on SPARK-2429: {quote} Can you add a breakdown of the timings fo

[jira] [Commented] (SPARK-3987) NNLS generates incorrect result

2014-10-22 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180001#comment-14180001 ] Debasish Das commented on SPARK-3987: - I will test it but this is how I called NNLS...

[jira] [Comment Edited] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

2014-10-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157605#comment-14157605 ] Guoqiang Li edited comment on SPARK-1405 at 10/22/14 3:28 PM: --

[jira] [Comment Edited] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

2014-10-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157605#comment-14157605 ] Guoqiang Li edited comment on SPARK-1405 at 10/22/14 3:30 PM: --

[jira] [Commented] (SPARK-3987) NNLS generates incorrect result

2014-10-22 Thread Shuo Xiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180073#comment-14180073 ] Shuo Xiang commented on SPARK-3987: --- [~debasish83] I think you are correct. Could you pl

[jira] [Created] (SPARK-4047) Generate runtime warning for naive implementation of PageRank example

2014-10-22 Thread Varadharajan (JIRA)
Varadharajan created SPARK-4047: --- Summary: Generate runtime warning for naive implementation of PageRank example Key: SPARK-4047 URL: https://issues.apache.org/jira/browse/SPARK-4047 Project: Spark

[jira] [Updated] (SPARK-4047) Generate runtime warning for naive implementation of PageRank example

2014-10-22 Thread Varadharajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varadharajan updated SPARK-4047: Description: Based on SPARK-2434, we're generating runtime warnings to denote that the exampl

[jira] [Commented] (SPARK-4047) Generate runtime warning for naive implementation of PageRank example

2014-10-22 Thread Varadharajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180083#comment-14180083 ] Varadharajan commented on SPARK-4047: - I'm working on this issue. > Generate runtime

[jira] [Comment Edited] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

2014-10-22 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157605#comment-14157605 ] Guoqiang Li edited comment on SPARK-1405 at 10/22/14 3:57 PM: --

[jira] [Commented] (SPARK-3359) `sbt/sbt unidoc` doesn't work with Java 8

2014-10-22 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180104#comment-14180104 ] holdenk commented on SPARK-3359: I think I've got a fix for it, I'll send a PR :) > `sbt/

[jira] [Resolved] (SPARK-3995) [PYSPARK] PySpark's sample methods do not work with NumPy 1.9

2014-10-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-3995. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2889 [https://githu

[jira] [Commented] (SPARK-4047) Generate runtime warning for naive implementation of PageRank example

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180147#comment-14180147 ] Apache Spark commented on SPARK-4047: - User 'varadharajan' has created a pull request

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-22 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180155#comment-14180155 ] koert kuipers commented on SPARK-3655: -- i am not sure repartitionAndSortWithinPartiti

[jira] [Comment Edited] (SPARK-3655) Secondary sort

2014-10-22 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180155#comment-14180155 ] koert kuipers edited comment on SPARK-3655 at 10/22/14 4:54 PM:

[jira] [Created] (SPARK-4048) Enhance and extend hadoop-provided profile

2014-10-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-4048: - Summary: Enhance and extend hadoop-provided profile Key: SPARK-4048 URL: https://issues.apache.org/jira/browse/SPARK-4048 Project: Spark Issue Type: Improv

[jira] [Updated] (SPARK-4048) Enhance and extend hadoop-provided profile

2014-10-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-4048: -- Description: The hadoop-provided profile is used to not package Hadoop dependencies inside the

[jira] [Commented] (SPARK-3987) NNLS generates incorrect result

2014-10-22 Thread Debasish Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180314#comment-14180314 ] Debasish Das commented on SPARK-3987: - [~coderxiang] changing to 1e-6 to 1e-7 fixes th

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-22 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180348#comment-14180348 ] koert kuipers commented on SPARK-3655: -- i went through the code. to allow a secondary

[jira] [Created] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2014-10-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4049: - Summary: Storage web UI "fraction cached" shows as > 100% Key: SPARK-4049 URL: https://issues.apache.org/jira/browse/SPARK-4049 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-22 Thread Tal Sliwowicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180512#comment-14180512 ] Tal Sliwowicz commented on SPARK-4006: -- Cool! Would be very interesting to know. For

[jira] [Comment Edited] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-22 Thread Tal Sliwowicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180512#comment-14180512 ] Tal Sliwowicz edited comment on SPARK-4006 at 10/22/14 8:48 PM:

[jira] [Created] (SPARK-4050) Caching fails for queries with sorts and projects

2014-10-22 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-4050: --- Summary: Caching fails for queries with sorts and projects Key: SPARK-4050 URL: https://issues.apache.org/jira/browse/SPARK-4050 Project: Spark Issue T

[jira] [Created] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-22 Thread Chris Grier (JIRA)
Chris Grier created SPARK-4051: -- Summary: Rows in python should support conversion to dictionary Key: SPARK-4051 URL: https://issues.apache.org/jira/browse/SPARK-4051 Project: Spark Issue Type:

[jira] [Updated] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-22 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4051: Affects Version/s: 1.1.0 > Rows in python should support conversion to dictionary >

[jira] [Updated] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-22 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4051: Target Version/s: 1.2.0 > Rows in python should support conversion to dictionary > -

[jira] [Updated] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-22 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4051: Assignee: Davies Liu > Rows in python should support conversion to dictionary >

[jira] [Commented] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180599#comment-14180599 ] Apache Spark commented on SPARK-4051: - User 'davies' has created a pull request for th

[jira] [Updated] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3877: - Target Version/s: 1.1.1, 1.2.0 Affects Version/s: 1.1.0 Fix Version/s: 1.2.0 Assi

[jira] [Updated] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3877: - Priority: Major (was: Minor) > The exit code of spark-submit is still 0 when an yarn application fails >

[jira] [Closed] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3877. Resolution: Fixed Fix Version/s: 1.1.1 > The exit code of spark-submit is still 0 when an yarn applic

[jira] [Commented] (SPARK-3877) The exit code of spark-submit is still 0 when an yarn application fails

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180628#comment-14180628 ] Apache Spark commented on SPARK-3877: - User 'zsxwing' has created a pull request for t

[jira] [Resolved] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3426. --- Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Fixed in 1.1.1. and 1.2.0 by my

[jira] [Resolved] (SPARK-3367) Remove spark.shuffle.spill.compress (replace it with existing spark.shuffle.compress)

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3367. --- Resolution: Won't Fix Resolving this as "Won't Fix" for now, given the discussion on that PR. We mig

[jira] [Assigned] (SPARK-2353) ArrayIndexOutOfBoundsException in scheduler

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-2353: - Assignee: Josh Rosen > ArrayIndexOutOfBoundsException in scheduler >

[jira] [Resolved] (SPARK-2353) ArrayIndexOutOfBoundsException in scheduler

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2353. --- Resolution: Fixed Fix Version/s: 1.1.0 This looks like a duplicate of SPARK-2931, which was fix

[jira] [Resolved] (SPARK-3709) Executors don't always report broadcast block removal properly back to the driver

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3709. --- Resolution: Fixed Fix Version/s: 1.0.3 1.2.0 1.1.1 It loo

[jira] [Updated] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empt

[jira] [Updated] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Description: {code} sc.makeRDD(0 until 10, 1000).repartition(2001).collect() {code} returns `Array()`.

[jira] [Commented] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180737#comment-14180737 ] Josh Rosen commented on SPARK-4019: --- This also explains another occurrence of the Snappy

[jira] [Created] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Yin Huai (JIRA)
Yin Huai created SPARK-4052: --- Summary: Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map) Key: SPARK-4052 URL: https://issues.apache.org/jira/browse/SPARK-40

[jira] [Updated] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-4052: Description: {code} val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) import sqlContext.createS

[jira] [Updated] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-4052: Description: Seems ScalaReflection and InsertIntoHiveTable only take scala.collection.immutable.Map as the

[jira] [Updated] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-4052: Description: Seems ScalaReflection and InsertIntoHiveTable only take scala.collection.immutable.Map as the

[jira] [Commented] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180749#comment-14180749 ] Yin Huai commented on SPARK-4052: - I searched our sql code base with {code} grep -r "typeO

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180758#comment-14180758 ] Josh Rosen commented on SPARK-3630: --- I found another cause: *Errors in reduce phases fo

[jira] [Commented] (SPARK-4052) Use scala.collection.Map for pattern matching instead of using Predef.Map (it is scala.collection.immutable.Map)

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180817#comment-14180817 ] Apache Spark commented on SPARK-4052: - User 'yhuai' has created a pull request for thi

[jira] [Created] (SPARK-4053) Block generator throttling in NetworkReceiverSuite is flaky

2014-10-22 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-4053: Summary: Block generator throttling in NetworkReceiverSuite is flaky Key: SPARK-4053 URL: https://issues.apache.org/jira/browse/SPARK-4053 Project: Spark Is

[jira] [Commented] (SPARK-4053) Block generator throttling in NetworkReceiverSuite is flaky

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180832#comment-14180832 ] Apache Spark commented on SPARK-4053: - User 'tdas' has created a pull request for this

[jira] [Updated] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-1239: -- Assignee: Josh Rosen (was: Kostas Sakellis) I'm re-assigning this to me since I've been working in this

[jira] [Commented] (SPARK-3988) Public API for DateType support

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180928#comment-14180928 ] Apache Spark commented on SPARK-3988: - User 'adrian-wang' has created a pull request f

[jira] [Commented] (SPARK-3988) Public API for DateType support

2014-10-22 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180933#comment-14180933 ] Adrian Wang commented on SPARK-3988: have to investigate solution 3 in spark-2179 > P

[jira] [Created] (SPARK-4054) Dead link in README

2014-10-22 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4054: - Summary: Dead link in README Key: SPARK-4054 URL: https://issues.apache.org/jira/browse/SPARK-4054 Project: Spark Issue Type: Bug Components: Doc

[jira] [Commented] (SPARK-4054) Dead link in README

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180946#comment-14180946 ] Apache Spark commented on SPARK-4054: - User 'sarutak' has created a pull request for t

[jira] [Resolved] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3812. Resolution: Fixed Assignee: Prashant Sharma Fixed by: https://github.com/apache/spark/

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

2014-10-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180992#comment-14180992 ] Ryan Williams commented on SPARK-4002: -- [~jerryshao] cool, a couple of notes: * If yo

[jira] [Updated] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

2014-10-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-4002: - Attachment: unit-tests.log unit-tests.log file from running {{mvn clean test -Dsuites='*KafkaStre

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

2014-10-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181006#comment-14181006 ] Saisai Shao commented on SPARK-4002: Thanks a lot Ryan for your detailed description,

[jira] [Updated] (SPARK-4002) KafkaStreamSuite "Kafka input stream" case fails on OSX

2014-10-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-4002: - Description: [~sowen] mentioned this on spark-dev [here|http://mail-archives.apache.org/mod_mbox/

[jira] [Updated] (SPARK-4002) KafkaStreamSuite "Kafka input stream" case fails on OSX

2014-10-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-4002: - Description: [~sowen] mentioned this on spark-dev [here|http://mail-archives.apache.org/mod_mbox/

[jira] [Closed] (SPARK-4054) Dead link in README

2014-10-22 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta closed SPARK-4054. - Resolution: Not a Problem > Dead link in README > --- > > Key: SPA

[jira] [Created] (SPARK-4055) Inconsistent spelling 'MLlib' and 'MLLib'

2014-10-22 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4055: - Summary: Inconsistent spelling 'MLlib' and 'MLLib' Key: SPARK-4055 URL: https://issues.apache.org/jira/browse/SPARK-4055 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 reducers may drop all data when partitions are mostly empty or ca

[jira] [Commented] (SPARK-4055) Inconsistent spelling 'MLlib' and 'MLLib'

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181034#comment-14181034 ] Apache Spark commented on SPARK-4055: - User 'sarutak' has created a pull request for t

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 map partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 map partitions may drop all data when partitions are mostly empty

[jira] [Created] (SPARK-4056) Upgrade snappy-java to 1.1.1.4

2014-10-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4056: - Summary: Upgrade snappy-java to 1.1.1.4 Key: SPARK-4056 URL: https://issues.apache.org/jira/browse/SPARK-4056 Project: Spark Issue Type: Improvement Re

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181044#comment-14181044 ] Josh Rosen commented on SPARK-3630: --- snappy-java just published a new release (1.1.1.4)

[jira] [Created] (SPARK-4057) Use -agentlib instead of -Xdebug in sbt--launch-lib.bash for debugging

2014-10-22 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4057: - Summary: Use -agentlib instead of -Xdebug in sbt--launch-lib.bash for debugging Key: SPARK-4057 URL: https://issues.apache.org/jira/browse/SPARK-4057 Project: Spar

[jira] [Updated] (SPARK-4057) Use -agentlib instead of -Xdebug in sbt-launch-lib.bash for debugging

2014-10-22 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-4057: -- Summary: Use -agentlib instead of -Xdebug in sbt-launch-lib.bash for debugging (was: Use -agen

[jira] [Commented] (SPARK-4057) Use -agentlib instead of -Xdebug in sbt-launch-lib.bash for debugging

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181049#comment-14181049 ] Apache Spark commented on SPARK-4057: - User 'sarutak' has created a pull request for t

[jira] [Issue Comment Deleted] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-22 Thread jeanlyn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-3967: --- Comment: was deleted (was: dsa dsa) > Spark applications fail in yarn-cluster mode when the directori

[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-22 Thread jeanlyn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181057#comment-14181057 ] jeanlyn commented on SPARK-3967: dsa dsa > Spark applications fail in yarn-cluste

[jira] [Created] (SPARK-4058) Log file name is hard coded even though there is a variable '$LOG_FILE '

2014-10-22 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4058: - Summary: Log file name is hard coded even though there is a variable '$LOG_FILE ' Key: SPARK-4058 URL: https://issues.apache.org/jira/browse/SPARK-4058 Project: Spa

[jira] [Commented] (SPARK-4058) Log file name is hard coded even though there is a variable '$LOG_FILE '

2014-10-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181060#comment-14181060 ] Apache Spark commented on SPARK-4058: - User 'sarutak' has created a pull request for t

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181066#comment-14181066 ] Patrick Wendell commented on SPARK-3655: Hey [~koertkuipers] - i'm not an expert o

[jira] [Created] (SPARK-4059) spark-master/spark-worker may use SPARK_MASTER_IP/STANDALONE_SPARK_MASTER_HOST

2014-10-22 Thread Guo Ruijing (JIRA)
Guo Ruijing created SPARK-4059: -- Summary: spark-master/spark-worker may use SPARK_MASTER_IP/STANDALONE_SPARK_MASTER_HOST Key: SPARK-4059 URL: https://issues.apache.org/jira/browse/SPARK-4059 Project: Spa

[jira] [Updated] (SPARK-4020) Failed executor not properly removed if it has not run tasks

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4020: --- Component/s: Spark Core > Failed executor not properly removed if it has not run tasks > -