[jira] [Commented] (SPARK-2982) Glitch of spark streaming

2014-08-11 Thread dai zhiyuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093816#comment-14093816 ] dai zhiyuan commented on SPARK-2982: [~srowen] Please see the attached file. > Glitch

[jira] [Commented] (SPARK-2981) PartitionStrategy: VertexID hash overflow

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093813#comment-14093813 ] Apache Spark commented on SPARK-2981: - User 'larryxiao' has created a pull request for

[jira] [Updated] (SPARK-2981) PartitionStrategy: VertexID hash overflow

2014-08-11 Thread Larry Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Xiao updated SPARK-2981: -- Description: In EdgePartition1D, a PartitionID is calculated by multiplying VertexId with a mixingPrim

[jira] [Updated] (SPARK-2982) Glitch of spark streaming

2014-08-11 Thread dai zhiyuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dai zhiyuan updated SPARK-2982: --- Attachment: network.png io.png cpu.png > Glitch of spark streaming >

[jira] [Updated] (SPARK-2982) Glitch of spark streaming

2014-08-11 Thread dai zhiyuan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dai zhiyuan updated SPARK-2982: --- Description: spark streaming task startup time is very focused,It creates a problem which is glitch o

[jira] [Commented] (SPARK-2982) Glitch of spark streaming

2014-08-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093794#comment-14093794 ] Sean Owen commented on SPARK-2982: -- I find it hard to understand the problem or solution

[jira] [Created] (SPARK-2985) BlockGenerator not available.

2014-08-11 Thread dai zhiyuan (JIRA)
dai zhiyuan created SPARK-2985: -- Summary: BlockGenerator not available. Key: SPARK-2985 URL: https://issues.apache.org/jira/browse/SPARK-2985 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-2650) Caching tables larger than memory causes OOMs

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093785#comment-14093785 ] Apache Spark commented on SPARK-2650: - User 'liancheng' has created a pull request for

[jira] [Updated] (SPARK-2984) FileNotFoundException on _temporary directory

2014-08-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2984: -- Description: We've seen several stacktraces and threads on the user mailing list where people are havi

[jira] [Created] (SPARK-2984) FileNotFoundException on _temporary directory

2014-08-11 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2984: - Summary: FileNotFoundException on _temporary directory Key: SPARK-2984 URL: https://issues.apache.org/jira/browse/SPARK-2984 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2983) improve performance of sortByKey()

2014-08-11 Thread Davies Liu (JIRA)
Davies Liu created SPARK-2983: - Summary: improve performance of sortByKey() Key: SPARK-2983 URL: https://issues.apache.org/jira/browse/SPARK-2983 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-2923) Implement some basic linalg operations in MLlib

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2923. -- Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 1849 [https://gith

[jira] [Commented] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-11 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093746#comment-14093746 ] Jianshi Huang commented on SPARK-2890: -- My use case: The result will be parsed into

[jira] [Created] (SPARK-2982) Glitch of spark streaming

2014-08-11 Thread dai zhiyuan (JIRA)
dai zhiyuan created SPARK-2982: -- Summary: Glitch of spark streaming Key: SPARK-2982 URL: https://issues.apache.org/jira/browse/SPARK-2982 Project: Spark Issue Type: Improvement Compone

[jira] [Resolved] (SPARK-2934) Adding LogisticRegressionWithLBFGS for training with LBFGS Optimizer

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-2934. -- Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 1862 [https://gith

[jira] [Resolved] (SPARK-2826) Reduce the Memory Copy for HashOuterJoin

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2826. - Resolution: Fixed Fix Version/s: 1.1.0 > Reduce the Memory Copy for HashOuterJoin

[jira] [Updated] (SPARK-2981) PartitionStrategy: VertexID hash overflow

2014-08-11 Thread Larry Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Xiao updated SPARK-2981: -- Description: In PartitionStrategy.scala a PartitionID is calculated by multiplying VertexId with a mix

[jira] [Created] (SPARK-2981) PartitionStrategy: VertexID hash overflow

2014-08-11 Thread Larry Xiao (JIRA)
Larry Xiao created SPARK-2981: - Summary: PartitionStrategy: VertexID hash overflow Key: SPARK-2981 URL: https://issues.apache.org/jira/browse/SPARK-2981 Project: Spark Issue Type: Bug C

[jira] [Resolved] (SPARK-2650) Caching tables larger than memory causes OOMs

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2650. - Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Michael Armbrust

[jira] [Resolved] (SPARK-2968) Fix nullabilities of Explode.

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2968. - Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Takuya Ueshin > Fix null

[jira] [Resolved] (SPARK-2965) Fix HashOuterJoin output nullabilities.

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2965. - Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Takuya Ueshin > Fix Hash

[jira] [Resolved] (SPARK-2590) Add config property to disable incremental collection used in Thrift server

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2590. - Resolution: Fixed Fix Version/s: 1.1.0 > Add config property to disable incrementa

[jira] [Resolved] (SPARK-2844) Existing JVM Hive Context not correctly used in Python Hive Context

2014-08-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-2844. - Resolution: Fixed Fix Version/s: 1.1.0 > Existing JVM Hive Context not correctly u

[jira] [Updated] (SPARK-2934) Adding LogisticRegressionWithLBFGS for training with LBFGS Optimizer

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2934: - Assignee: DB Tsai > Adding LogisticRegressionWithLBFGS for training with LBFGS Optimizer >

[jira] [Updated] (SPARK-2980) Python support for chi-squared test

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2980: - Assignee: (was: Doris Xin) > Python support for chi-squared test > --

[jira] [Created] (SPARK-2980) Python support for chi-squared test

2014-08-11 Thread Doris Xin (JIRA)
Doris Xin created SPARK-2980: Summary: Python support for chi-squared test Key: SPARK-2980 URL: https://issues.apache.org/jira/browse/SPARK-2980 Project: Spark Issue Type: Sub-task Comp

[jira] [Updated] (SPARK-2515) Chi-squared test

2014-08-11 Thread Doris Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doris Xin updated SPARK-2515: - Summary: Chi-squared test (was: Hypothesis testing) > Chi-squared test > > >

[jira] [Closed] (SPARK-2515) Chi-squared test

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-2515. Resolution: Implemented Target Version/s: 1.1.0 > Chi-squared test > > >

[jira] [Updated] (SPARK-2515) Hypothesis testing

2014-08-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-2515: - Fix Version/s: 1.1.0 > Hypothesis testing > -- > > Key: SPARK-251

[jira] [Updated] (SPARK-2979) Improve the convergence rate by minimizing the condition number in LOR with LBFGS

2014-08-11 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-2979: --- Summary: Improve the convergence rate by minimizing the condition number in LOR with LBFGS (was: Improve the

[jira] [Commented] (SPARK-2979) Improve the convergence rate by minimize the condition number in LOR with LBFGS

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093604#comment-14093604 ] Apache Spark commented on SPARK-2979: - User 'dbtsai' has created a pull request for th

[jira] [Created] (SPARK-2979) Improve the convergence rate by minimize the condition number in LOR with LBFGS

2014-08-11 Thread DB Tsai (JIRA)
DB Tsai created SPARK-2979: -- Summary: Improve the convergence rate by minimize the condition number in LOR with LBFGS Key: SPARK-2979 URL: https://issues.apache.org/jira/browse/SPARK-2979 Project: Spark

[jira] [Updated] (SPARK-2978) Provide an MR-style shuffle transformation

2014-08-11 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-2978: -- Description: For Hive on Spark joins in particular, and for running legacy MR code in general, I think

[jira] [Updated] (SPARK-2978) Provide an MR-style shuffle transformation

2014-08-11 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-2978: -- Description: For Hive on Spark joins in particular, and for running legacy MR code in general, I think

[jira] [Updated] (SPARK-2978) Provide an MR-style shuffle transformation

2014-08-11 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-2978: -- Description: For Hive on Spark joins in particular, and for running legacy MR code in general, I think

[jira] [Created] (SPARK-2978) Provide an MR-style shuffle transformation

2014-08-11 Thread Sandy Ryza (JIRA)
Sandy Ryza created SPARK-2978: - Summary: Provide an MR-style shuffle transformation Key: SPARK-2978 URL: https://issues.apache.org/jira/browse/SPARK-2978 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-2975) SPARK_LOCAL_DIRS may cause problems when running in local mode

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2975: -- Priority: Critical (was: Minor) I'm raising the priority of this issue to 'critical', since it causes

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093468#comment-14093468 ] Sean Owen commented on SPARK-1297: -- Yes I think you'd need to reflect that in changes to

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093466#comment-14093466 ] Ted Yu commented on SPARK-1297: --- w.r.t. build, by default, hbase-hadoop1 would be used. If u

[jira] [Commented] (SPARK-1065) PySpark runs out of memory with large broadcast variables

2014-08-11 Thread Vlad Frolov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093413#comment-14093413 ] Vlad Frolov commented on SPARK-1065: I am facing the same issue in my project, where I

[jira] [Commented] (SPARK-2931) getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsException

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093295#comment-14093295 ] Apache Spark commented on SPARK-2931: - User 'JoshRosen' has created a pull request for

[jira] [Comment Edited] (SPARK-2891) Daemon failed to launch worker

2014-08-11 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093265#comment-14093265 ] Davies Liu edited comment on SPARK-2891 at 8/11/14 8:45 PM: du

[jira] [Resolved] (SPARK-2891) Daemon failed to launch worker

2014-08-11 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-2891. --- Resolution: Duplicate Fix Version/s: 1.1.0 duplicated to 2898 > Daemon failed to launch worke

[jira] [Commented] (SPARK-1284) pyspark hangs after IOError on Executor

2014-08-11 Thread Jim Blomo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093219#comment-14093219 ] Jim Blomo commented on SPARK-1284: -- I will try to reproduce on the 1.1 branch later this

[jira] [Updated] (SPARK-2420) Dependency changes for compatibility with Hive

2014-08-11 Thread Brock Noland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated SPARK-2420: Labels: Hive (was: ) > Dependency changes for compatibility with Hive > --

[jira] [Commented] (SPARK-2976) There are too many tabs in some source files

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093175#comment-14093175 ] Apache Spark commented on SPARK-2976: - User 'sarutak' has created a pull request for t

[jira] [Resolved] (SPARK-2101) Python unit tests fail on Python 2.6 because of lack of unittest.skipIf()

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2101. --- Resolution: Fixed Fix Version/s: 1.1.0 > Python unit tests fail on Python 2.6 because of lack

[jira] [Created] (SPARK-2977) Fix handling of short shuffle manager names in ShuffleBlockManager

2014-08-11 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-2977: - Summary: Fix handling of short shuffle manager names in ShuffleBlockManager Key: SPARK-2977 URL: https://issues.apache.org/jira/browse/SPARK-2977 Project: Spark I

[jira] [Resolved] (SPARK-2910) Test with Python 2.6 on Jenkins

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2910. --- > Test with Python 2.6 on Jenkins > --- > > Key: SPARK-2910 >

[jira] [Resolved] (SPARK-2954) PySpark MLlib serialization tests fail on Python 2.6

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2954. --- Resolution: Fixed Fix Version/s: 1.1.0 > PySpark MLlib serialization tests fail on Python 2.6

[jira] [Resolved] (SPARK-2948) PySpark doesn't work on Python 2.6

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2948. --- Resolution: Fixed Fix Version/s: 1.1.0 > PySpark doesn't work on Python 2.6 >

[jira] [Commented] (SPARK-2700) Hidden files (such as .impala_insert_staging) should be filtered out by sqlContext.parquetFile

2014-08-11 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093150#comment-14093150 ] Yin Huai commented on SPARK-2700: - Can we resolve it? > Hidden files (such as .impala_ins

[jira] [Commented] (SPARK-2790) PySpark zip() doesn't work properly if RDDs have different serializers

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093133#comment-14093133 ] Apache Spark commented on SPARK-2790: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-1284) pyspark hangs after IOError on Executor

2014-08-11 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093137#comment-14093137 ] Davies Liu commented on SPARK-1284: --- [~jblomo], could you reproduce this on master or 1.

[jira] [Commented] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-11 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093119#comment-14093119 ] Yin Huai commented on SPARK-2890: - What is the semantic when you have columns with same na

[jira] [Created] (SPARK-2976) There are too many tabs in some source files

2014-08-11 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2976: - Summary: There are too many tabs in some source files Key: SPARK-2976 URL: https://issues.apache.org/jira/browse/SPARK-2976 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-2963) The description about building to use HiveServer and CLI is incomplete

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2963: -- Summary: The description about building to use HiveServer and CLI is incomplete (was: The desc

[jira] [Updated] (SPARK-2931) getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsException

2014-08-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2931: --- Fix Version/s: (was: 1.1.0) > getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsExcep

[jira] [Updated] (SPARK-2931) getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsException

2014-08-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2931: --- Target Version/s: 1.1.0 > getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsException > -

[jira] [Created] (SPARK-2975) SPARK_LOCAL_DIRS may cause problems when running in local mode

2014-08-11 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-2975: - Summary: SPARK_LOCAL_DIRS may cause problems when running in local mode Key: SPARK-2975 URL: https://issues.apache.org/jira/browse/SPARK-2975 Project: Spark Issue

[jira] [Updated] (SPARK-2717) BasicBlockFetchIterator#next should log when it gets stuck

2014-08-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2717: --- Priority: Critical (was: Major) > BasicBlockFetchIterator#next should log when it gets stuck

[jira] [Updated] (SPARK-2717) BasicBlockFetchIterator#next should log when it gets stuck

2014-08-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2717: --- Priority: Major (was: Blocker) > BasicBlockFetchIterator#next should log when it gets stuck

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093018#comment-14093018 ] Apache Spark commented on SPARK-1297: - User 'tedyu' has created a pull request for thi

[jira] [Created] (SPARK-2974) Utils.getLocalDir() may return non-existent spark.local.dir directory

2014-08-11 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-2974: - Summary: Utils.getLocalDir() may return non-existent spark.local.dir directory Key: SPARK-2974 URL: https://issues.apache.org/jira/browse/SPARK-2974 Project: Spark

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093012#comment-14093012 ] Ted Yu commented on SPARK-1297: --- https://github.com/apache/spark/pull/1893 > Upgrade HBase

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092988#comment-14092988 ] Ted Yu commented on SPARK-1297: --- HBase client doesn't need to specify dependency on hbase-ha

[jira] [Created] (SPARK-2973) Add a way to show tables without executing a job

2014-08-11 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2973: - Summary: Add a way to show tables without executing a job Key: SPARK-2973 URL: https://issues.apache.org/jira/browse/SPARK-2973 Project: Spark Issue Type:

[jira] [Created] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

2014-08-11 Thread Shay Rojansky (JIRA)
Shay Rojansky created SPARK-2972: Summary: APPLICATION_COMPLETE not created in Python unless context explicitly stopped Key: SPARK-2972 URL: https://issues.apache.org/jira/browse/SPARK-2972 Project: S

[jira] [Commented] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092967#comment-14092967 ] Sean Owen commented on SPARK-1297: -- I think you may want to open a PR rather than post pa

[jira] [Commented] (SPARK-2931) getAllowedLocalityLevel() throws ArrayIndexOutOfBoundsException

2014-08-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092966#comment-14092966 ] Josh Rosen commented on SPARK-2931: --- Thanks for investigating and reproducing this issue

[jira] [Updated] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-08-11 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated SPARK-1297: -- Attachment: spark-1297-v4.txt Patch v4 adds two profiles to examples/pom.xml : hbase-hadoop1 (default) hbase-h

[jira] [Updated] (SPARK-2963) The description about building to use HiveServer and CLI is imcomplete

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2963: -- Description: Currently, if we'd like to use HiveServer or CLI for SparkSQL, we need to use -Phi

[jira] [Updated] (SPARK-2963) The description about building to use HiveServer and CLI is imcomplete

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2963: -- Summary: The description about building to use HiveServer and CLI is imcomplete (was: There no

[jira] [Commented] (SPARK-2963) The description about building to use HiveServer and CLI is imcomplete

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092894#comment-14092894 ] Kousuke Saruta commented on SPARK-2963: --- I've updated this title and Github's one.

[jira] [Comment Edited] (SPARK-2963) There no documentation about building to use HiveServer and CLI for SparkSQL

2014-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092889#comment-14092889 ] Cheng Lian edited comment on SPARK-2963 at 8/11/14 3:31 PM: Ac

[jira] [Commented] (SPARK-2963) There no documentation about building to use HiveServer and CLI for SparkSQL

2014-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092889#comment-14092889 ] Cheng Lian commented on SPARK-2963: --- Actually [there is|https://github.com/apache/spark

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2014-08-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092881#comment-14092881 ] Thomas Graves commented on SPARK-2089: -- Sandy, just wondering if you have any ETA on

[jira] [Commented] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092879#comment-14092879 ] Kousuke Saruta commented on SPARK-2970: --- [~liancheng] Thank you pointing my mistake.

[jira] [Updated] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-2970: -- Description: When spark-sql script run with spark.eventLog.enabled set true, it ends with IOEx

[jira] [Commented] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092860#comment-14092860 ] Cheng Lian commented on SPARK-2970: --- [~sarutak] Would you mind to update the issue descr

[jira] [Commented] (SPARK-1777) Pass "cached" blocks directly to disk if memory is not large enough

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092826#comment-14092826 ] Apache Spark commented on SPARK-1777: - User 'liyezhang556520' has created a pull reque

[jira] [Commented] (SPARK-2962) Suboptimal scheduling in spark

2014-08-11 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092807#comment-14092807 ] Mridul Muralidharan commented on SPARK-2962: On further investigation : a) Th

[jira] [Created] (SPARK-2971) Orphaned YARN ApplicationMaster lingers forever

2014-08-11 Thread Shay Rojansky (JIRA)
Shay Rojansky created SPARK-2971: Summary: Orphaned YARN ApplicationMaster lingers forever Key: SPARK-2971 URL: https://issues.apache.org/jira/browse/SPARK-2971 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092710#comment-14092710 ] Apache Spark commented on SPARK-2970: - User 'sarutak' has created a pull request for t

[jira] [Commented] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092705#comment-14092705 ] Kousuke Saruta commented on SPARK-2970: --- I noticed it's not caused by the reason abo

[jira] [Created] (SPARK-2970) spark-sql script ends with IOException when EventLogging is enabled

2014-08-11 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-2970: - Summary: spark-sql script ends with IOException when EventLogging is enabled Key: SPARK-2970 URL: https://issues.apache.org/jira/browse/SPARK-2970 Project: Spark

[jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation with custom Kryo Registrator

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092677#comment-14092677 ] Apache Spark commented on SPARK-2878: - User 'GrahamDennis' has created a pull request

[jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation with custom Kryo Registrator

2014-08-11 Thread Graham Dennis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092675#comment-14092675 ] Graham Dennis commented on SPARK-2878: -- I've created a pull request with work-in-prog

[jira] [Commented] (SPARK-2969) Make ScalaReflection be able to handle MapType.containsNull and MapType.valueContainsNull.

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092671#comment-14092671 ] Apache Spark commented on SPARK-2969: - User 'ueshin' has created a pull request for th

[jira] [Updated] (SPARK-2969) Make ScalaReflection be able to handle MapType.containsNull and MapType.valueContainsNull.

2014-08-11 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-2969: - Description: Make {{ScalaReflection}} be able to handle like: - Seq\[Int] as ArrayType(IntegerTy

[jira] [Created] (SPARK-2969) Make ScalaReflection be able to handle MapType.containsNull and MapType.valueContainsNull.

2014-08-11 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-2969: Summary: Make ScalaReflection be able to handle MapType.containsNull and MapType.valueContainsNull. Key: SPARK-2969 URL: https://issues.apache.org/jira/browse/SPARK-2969

[jira] [Commented] (SPARK-2968) Fix nullabilities of Explode.

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092645#comment-14092645 ] Apache Spark commented on SPARK-2968: - User 'ueshin' has created a pull request for th

[jira] [Created] (SPARK-2968) Fix nullabilities of Explode.

2014-08-11 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-2968: Summary: Fix nullabilities of Explode. Key: SPARK-2968 URL: https://issues.apache.org/jira/browse/SPARK-2968 Project: Spark Issue Type: Bug Compone

[jira] [Created] (SPARK-2967) Several SQL unit test failed when sort-based shuffle is enabled

2014-08-11 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-2967: -- Summary: Several SQL unit test failed when sort-based shuffle is enabled Key: SPARK-2967 URL: https://issues.apache.org/jira/browse/SPARK-2967 Project: Spark Is

[jira] [Updated] (SPARK-2966) Add an approximation algorithm for hierarchical clustering to MLlib

2014-08-11 Thread Yu Ishikawa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Ishikawa updated SPARK-2966: --- Summary: Add an approximation algorithm for hierarchical clustering to MLlib (was: Add an approximat

[jira] [Created] (SPARK-2966) Add an approximation algorithm for hierarchical clustering algorithm to MLlib

2014-08-11 Thread Yu Ishikawa (JIRA)
Yu Ishikawa created SPARK-2966: -- Summary: Add an approximation algorithm for hierarchical clustering algorithm to MLlib Key: SPARK-2966 URL: https://issues.apache.org/jira/browse/SPARK-2966 Project: Spar

[jira] [Updated] (SPARK-2862) DoubleRDDFunctions.histogram() throws exception for some inputs

2014-08-11 Thread Chandan Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandan Kumar updated SPARK-2862: - Description: histogram method call throws an IndexOutOfBoundsException when the choice of bucket

[jira] [Commented] (SPARK-2965) Fix HashOuterJoin output nullabilities.

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092584#comment-14092584 ] Apache Spark commented on SPARK-2965: - User 'ueshin' has created a pull request for th

[jira] [Created] (SPARK-2965) Fix HashOuterJoin output nullabilities.

2014-08-11 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-2965: Summary: Fix HashOuterJoin output nullabilities. Key: SPARK-2965 URL: https://issues.apache.org/jira/browse/SPARK-2965 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2964) Wrong silent option in spark-sql script

2014-08-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092515#comment-14092515 ] Apache Spark commented on SPARK-2964: - User 'sarutak' has created a pull request for t

  1   2   >