[jira] [Created] (SPARK-2191) Double execution with CREATE TABLE AS SELECT

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2191: --- Summary: Double execution with CREATE TABLE AS SELECT Key: SPARK-2191 URL: https://issues.apache.org/jira/browse/SPARK-2191 Project: Spark Issue Type:

[jira] [Created] (SPARK-2190) Specialized ColumnType for Timestamp

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2190: --- Summary: Specialized ColumnType for Timestamp Key: SPARK-2190 URL: https://issues.apache.org/jira/browse/SPARK-2190 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2189) Method for removing temp tables created by registerAsTable

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2189: --- Summary: Method for removing temp tables created by registerAsTable Key: SPARK-2189 URL: https://issues.apache.org/jira/browse/SPARK-2189 Project: Spark

[jira] [Commented] (SPARK-2155) Support effectful / non-deterministic key expressions in CASE WHEN statements

2014-06-18 Thread Zongheng Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036999#comment-14036999 ] Zongheng Yang commented on SPARK-2155: -- Sure, here is the link I used: https://cwiki

[jira] [Commented] (SPARK-2155) Support effectful / non-deterministic key expressions in CASE WHEN statements

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036992#comment-14036992 ] Reynold Xin commented on SPARK-2155: Can you include a link to some page that explains

[jira] [Commented] (SPARK-2180) HiveQL doesn't support GROUP BY with HAVING clauses

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036994#comment-14036994 ] Reynold Xin commented on SPARK-2180: This would be great to have, [~willbenton]. > Hi

[jira] [Commented] (SPARK-2136) Spark SQL does not disply the job description on web ui/ event log

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036987#comment-14036987 ] Reynold Xin commented on SPARK-2136: We should definitely do this for a SQL command li

[jira] [Resolved] (SPARK-2187) Explain command should not run the optimizer twice

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2187. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 > Explain command should not

[jira] [Updated] (SPARK-2183) Avoid loading/shuffling data twice in self-join query

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2183: --- Priority: Minor (was: Major) > Avoid loading/shuffling data twice in self-join query > -

[jira] [Updated] (SPARK-2179) Public API for DataTypes and Schema

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2179: --- Assignee: Yin Huai > Public API for DataTypes and Schema > --- > >

[jira] [Created] (SPARK-2188) Support sbt/sbt for Windows

2014-06-18 Thread Pat McDonough (JIRA)
Pat McDonough created SPARK-2188: Summary: Support sbt/sbt for Windows Key: SPARK-2188 URL: https://issues.apache.org/jira/browse/SPARK-2188 Project: Spark Issue Type: New Feature C

[jira] [Resolved] (SPARK-1964) Timestamp missing from HiveMetastore types parser

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-1964. Resolution: Fixed > Timestamp missing from HiveMetastore types parser > ---

[jira] [Issue Comment Deleted] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Jin updated SPARK-1112: Comment: was deleted (was: Awesome, looking forward to the fix. ) > When spark.akka.frameSize > 10, task r

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036903#comment-14036903 ] Chen Jin commented on SPARK-2156: - [~pwendell] [~mengxr][~gq], Thanks a lot, I will give i

[jira] [Commented] (SPARK-2138) The KMeans algorithm in the MLlib can lead to the Serialized Task size become bigger and bigger

2014-06-18 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036892#comment-14036892 ] DjvuLee commented on SPARK-2138: Thanks very much! I am very glad to see that I reported

[jira] [Commented] (SPARK-2138) The KMeans algorithm in the MLlib can lead to the Serialized Task size become bigger and bigger

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036891#comment-14036891 ] Xiangrui Meng commented on SPARK-2138: -- Backend doesn't read spark.akka.frameSize. >

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036874#comment-14036874 ] Patrick Wendell commented on SPARK-2156: I closed my patch in favor of Xiangrui's.

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036869#comment-14036869 ] Xiangrui Meng commented on SPARK-1112: -- PR: https://github.com/apache/spark/pull/1124

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036871#comment-14036871 ] Xiangrui Meng commented on SPARK-2156: -- PR: https://github.com/apache/spark/pull/1124

[jira] [Updated] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2156: --- Priority: Blocker (was: Critical) > When the size of serialized results for one partition is

[jira] [Updated] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2156: --- Target Version/s: 0.9.2, 1.0.1 (was: 1.0.1) > When the size of serialized results for one pa

[jira] [Updated] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2156: --- Fix Version/s: (was: 1.0.1) > When the size of serialized results for one partition is sl

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036866#comment-14036866 ] Patrick Wendell commented on SPARK-2156: [~xiaocai] Would you mind trying with the

[jira] [Assigned] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-2156: Assignee: Xiangrui Meng (was: Patrick Wendell) > When the size of serialized results for o

[jira] [Assigned] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reassigned SPARK-2156: -- Assignee: Patrick Wendell > When the size of serialized results for one partition is sl

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036863#comment-14036863 ] Chen Jin commented on SPARK-2156: - Yes, I am calling collect() at the end. > When the si

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036850#comment-14036850 ] Patrick Wendell commented on SPARK-2156: [~xiaocai] Are you calling collect() on t

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036849#comment-14036849 ] Patrick Wendell commented on SPARK-2156: Ah I see - thanks! I'm going to re-open i

[jira] [Reopened] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2156: > When the size of serialized results for one partition is slightly smaller > than 10MB (the d

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036843#comment-14036843 ] Chen Jin commented on SPARK-1112: - Awesome, looking forward to the fix. At least better er

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036844#comment-14036844 ] Guoqiang Li commented on SPARK-2156: [~pwendell] This seems to is not the same as

[jira] [Issue Comment Deleted] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-2156: --- Comment: was deleted (was: [~pwend...@gmail.com]) > When the size of serialized results for one part

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036838#comment-14036838 ] Guoqiang Li commented on SPARK-2156: [~pwend...@gmail.com] > When the size of seriali

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036840#comment-14036840 ] Chen Jin commented on SPARK-1112: - Awesome, looking forward to the fix. > When spark.akk

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036822#comment-14036822 ] Chen Jin commented on SPARK-2156: - [~gq] Thanks a lot for looking into this. The differenc

[jira] [Assigned] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-1112: Assignee: Xiangrui Meng > When spark.akka.frameSize > 10, task results bigger than 10MiB bl

[jira] [Updated] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1112: --- Fix Version/s: (was: 0.9.2) > When spark.akka.frameSize > 10, task results bigger than 10

[jira] [Resolved] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2156. Resolution: Duplicate > When the size of serialized results for one partition is slightly s

[jira] [Updated] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1112: --- Priority: Blocker (was: Critical) > When spark.akka.frameSize > 10, task results bigger than

[jira] [Updated] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1112: --- Target Version/s: 0.9.2, 1.0.1 > When spark.akka.frameSize > 10, task results bigger than 10M

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036801#comment-14036801 ] Patrick Wendell commented on SPARK-1112: We were able to reproduce this - thanks f

[jira] [Reopened] (SPARK-1209) SparkHadoopUtil should not use package org.apache.hadoop

2014-06-18 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened SPARK-1209: --- It doesn't look like this was actually fixed. > SparkHadoopUtil should not use package org.apache.hadoop

[jira] [Created] (SPARK-2187) Explain command should not run the optimizer twice

2014-06-18 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-2187: -- Summary: Explain command should not run the optimizer twice Key: SPARK-2187 URL: https://issues.apache.org/jira/browse/SPARK-2187 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-2184) AddExchange isn't idempotent

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2184. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 > AddExchange isn't idempote

[jira] [Created] (SPARK-2186) Spark SQL DSL support for simple aggregations such as SUM and AVG

2014-06-18 Thread Zongheng Yang (JIRA)
Zongheng Yang created SPARK-2186: Summary: Spark SQL DSL support for simple aggregations such as SUM and AVG Key: SPARK-2186 URL: https://issues.apache.org/jira/browse/SPARK-2186 Project: Spark

[jira] [Updated] (SPARK-2186) Spark SQL DSL support for simple aggregations such as SUM and AVG

2014-06-18 Thread Zongheng Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zongheng Yang updated SPARK-2186: - Affects Version/s: 1.0.0 > Spark SQL DSL support for simple aggregations such as SUM and AVG > --

[jira] [Created] (SPARK-2185) Scheduler should print warning when tasks are larger than the recommended size

2014-06-18 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-2185: - Summary: Scheduler should print warning when tasks are larger than the recommended size Key: SPARK-2185 URL: https://issues.apache.org/jira/browse/SPARK-2185 Projec

[jira] [Closed] (SPARK-2162) possible to read from removed block in blockmanager

2014-06-18 Thread Raymond Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Liu closed SPARK-2162. -- patch merged > possible to read from removed block in blockmanager > -

[jira] [Commented] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036704#comment-14036704 ] Chen Jin commented on SPARK-1112: - I have filed a bug https://issues.apache.org/jira/brows

[jira] [Updated] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

2014-06-18 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-1112: - Affects Version/s: 1.0.0 > When spark.akka.frameSize > 10, task results bigger than 10MiB block e

[jira] [Updated] (SPARK-2184) AddExchange isn't idempotent

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2184: --- Assignee: Michael Armbrust > AddExchange isn't idempotent > > >

[jira] [Created] (SPARK-2184) AddExchange isn't idempotent

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2184: --- Summary: AddExchange isn't idempotent Key: SPARK-2184 URL: https://issues.apache.org/jira/browse/SPARK-2184 Project: Spark Issue Type: Bug Co

[jira] [Created] (SPARK-2183) Avoid loading/shuffling data twice in self-join query

2014-06-18 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-2183: -- Summary: Avoid loading/shuffling data twice in self-join query Key: SPARK-2183 URL: https://issues.apache.org/jira/browse/SPARK-2183 Project: Spark Issue Type: I

[jira] [Updated] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Chen Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Jin updated SPARK-2156: Description: I have done some experiments when the frameSize is around 10MB . 1) spark.akka.frameSize = 1

[jira] [Commented] (SPARK-2121) Not fully cached when there is enough memory

2014-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036565#comment-14036565 ] Xiangrui Meng commented on SPARK-2121: -- Had an offline discussion with [~coderxiang].

[jira] [Updated] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-06-18 Thread Doris Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doris Xin updated SPARK-2182: - Attachment: Screen Shot 2014-06-18 at 3.28.44 PM.png How I spotted it in Eclipse > Scalastyle rule block

[jira] [Commented] (SPARK-914) Make RDD implement Scala and Java Iterable interfaces

2014-06-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036556#comment-14036556 ] Andrew Ash commented on SPARK-914: -- Please close this ticket -- it is a duplicate of SPARK

[jira] [Updated] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-06-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2182: -- Summary: Scalastyle rule blocking unicode operators (was: Scalastyle rule for unicode operators) > Sc

[jira] [Commented] (SPARK-2182) Scalastyle rule blocking unicode operators

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036550#comment-14036550 ] Reynold Xin commented on SPARK-2182: It was Eclipse. > Scalastyle rule blocking unic

[jira] [Created] (SPARK-2182) Scalastyle rule for unicode operators

2014-06-18 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-2182: - Summary: Scalastyle rule for unicode operators Key: SPARK-2182 URL: https://issues.apache.org/jira/browse/SPARK-2182 Project: Spark Issue Type: Bug Compo

[jira] [Commented] (SPARK-1466) Pyspark doesn't check if gateway process launches correctly

2014-06-18 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036316#comment-14036316 ] Kay Ousterhout commented on SPARK-1466: --- [~pwendell] I didn't merge this into the 1.

[jira] [Resolved] (SPARK-1466) Pyspark doesn't check if gateway process launches correctly

2014-06-18 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout resolved SPARK-1466. --- Resolution: Fixed Fix Version/s: (was: 1.0.1) 1.1.0 > Pyspark d

[jira] [Updated] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2104: --- Description: Otherwise it is pretty annoying to do a sort on types that are not java serializable.

[jira] [Updated] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2104: --- Description: Otherwise it is pretty annoying to do a sort on types that are not java serializable.

[jira] [Updated] (SPARK-768) Fail a task when the remote block it is fetching is not serializable

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-768: -- Target Version/s: 1.1.0 > Fail a task when the remote block it is fetching is not serializable > ---

[jira] [Resolved] (SPARK-2162) possible to read from removed block in blockmanager

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2162. Resolution: Fixed Fix Version/s: 1.1.0 Assignee: Raymond Liu > possible to read fro

[jira] [Commented] (SPARK-2177) describe table result contains only one column

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036207#comment-14036207 ] Yin Huai commented on SPARK-2177: - We may want to ask Hive to use JsonMetaDataFormatter fo

[jira] [Commented] (SPARK-2177) describe table result contains only one column

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036199#comment-14036199 ] Yin Huai commented on SPARK-2177: - I think we need a better way to handle the result of a

[jira] [Updated] (SPARK-2038) Don't shadow "conf" variable in saveAsHadoop functions

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2038: --- Labels: api-breaking (was: ) > Don't shadow "conf" variable in saveAsHadoop functions >

[jira] [Resolved] (SPARK-2176) Extra unnecessary exchange operator in the result of an explain command

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-2176. Resolution: Fixed Fix Version/s: 1.1.0 1.0.1 > Extra unnecessary exchange

[jira] [Created] (SPARK-2181) The keys for sorting the columns of Executor page in SparkUI are incorrect

2014-06-18 Thread Shuo Xiang (JIRA)
Shuo Xiang created SPARK-2181: - Summary: The keys for sorting the columns of Executor page in SparkUI are incorrect Key: SPARK-2181 URL: https://issues.apache.org/jira/browse/SPARK-2181 Project: Spark

[jira] [Updated] (SPARK-2176) Extra unnecessary exchange operator in the result of an explain command

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-2176: Summary: Extra unnecessary exchange operator in the result of an explain command (was: Extra unnecessary e

[jira] [Updated] (SPARK-2176) extra unnecessary exchange operator in the result of a explain command

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-2176: Summary: extra unnecessary exchange operator in the result of a explain command (was: extra unnecessary ex

[jira] [Updated] (SPARK-2176) Extra unnecessary exchange operator in the result of a explain command

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-2176: Summary: Extra unnecessary exchange operator in the result of a explain command (was: extra unnecessary ex

[jira] [Commented] (SPARK-2176) extra unnecessary exchange operator in group by

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035941#comment-14035941 ] Yin Huai commented on SPARK-2176: - OK. Let me explain the cause of this bug. When we crea

[jira] [Created] (SPARK-2180) HiveQL doesn't support GROUP BY with HAVING clauses

2014-06-18 Thread William Benton (JIRA)
William Benton created SPARK-2180: - Summary: HiveQL doesn't support GROUP BY with HAVING clauses Key: SPARK-2180 URL: https://issues.apache.org/jira/browse/SPARK-2180 Project: Spark Issue Typ

[jira] [Commented] (SPARK-2180) HiveQL doesn't support GROUP BY with HAVING clauses

2014-06-18 Thread William Benton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035907#comment-14035907 ] William Benton commented on SPARK-2180: --- (I'm working on a fix and will submit a PR

[jira] [Updated] (SPARK-2176) extra unnecessary exchange operator in group by

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-2176: Description: {code} hql("explain select * from src group by key").collect().foreach(println) [ExplainComma

[jira] [Commented] (SPARK-2176) extra unnecessary exchange operator in group by

2014-06-18 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035887#comment-14035887 ] Yin Huai commented on SPARK-2176: - Seems it is a bug related to Explain command. When I us

[jira] [Commented] (SPARK-1906) spark-submit doesn't send master URL to Driver in standalone cluster mode

2014-06-18 Thread Jacob Eisinger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035823#comment-14035823 ] Jacob Eisinger commented on SPARK-1906: --- A quick way to test is to run the the examp

[jira] [Updated] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-2156: --- Affects Version/s: 0.9.1 > When the size of serialized results for one partition is slightly smaller

[jira] [Commented] (SPARK-2156) When the size of serialized results for one partition is slightly smaller than 10MB (the default akka.frameSize), the execution blocks

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035795#comment-14035795 ] Guoqiang Li commented on SPARK-2156: The problem here: [Executor.scala#L215|https://g

[jira] [Updated] (SPARK-1471) Worker not recognize Driver state at standalone mode

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-1471: --- Fix Version/s: (was: 1.0.1) > Worker not recognize Driver state at standalone mode > ---

[jira] [Updated] (SPARK-2171) Groovy bindings for Spark

2014-06-18 Thread Artur Andrzejak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Andrzejak updated SPARK-2171: --- Attachment: Groovy4Spark starter kit.zip Added Groovy4Spark starter kit.zip: Example and ins

[jira] [Commented] (SPARK-2157) Can't write tight firewall rules for Spark

2014-06-18 Thread Egor Pahomov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035554#comment-14035554 ] Egor Pahomov commented on SPARK-2157: - Yep, I used patched version of spark. > Can't

[jira] [Commented] (SPARK-1353) IllegalArgumentException when writing to disk

2014-06-18 Thread jackielihf (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035543#comment-14035543 ] jackielihf commented on SPARK-1353: --- 2014-06-16 12:27:30,910 WARN [Result resolver threa

[jira] [Commented] (SPARK-1291) Link the spark UI to RM ui in yarn-client mode

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035522#comment-14035522 ] Guoqiang Li commented on SPARK-1291: [~rahulsinghal.iitd] Thank you, looking forward t

[jira] [Commented] (SPARK-1291) Link the spark UI to RM ui in yarn-client mode

2014-06-18 Thread Rahul Singhal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035512#comment-14035512 ] Rahul Singhal commented on SPARK-1291: -- [~gq] I was also able to get this working but

[jira] [Commented] (SPARK-2179) Public API for DataTypes and Schema

2014-06-18 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035501#comment-14035501 ] Michael Armbrust commented on SPARK-2179: - We need to nail down array and map null

[jira] [Commented] (SPARK-1291) Link the spark UI to RM ui in yarn-client mode

2014-06-18 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035467#comment-14035467 ] Guoqiang Li commented on SPARK-1291: PR: https://github.com/apache/spark/pull/1112 >

[jira] [Created] (SPARK-2179) Public API for DataTypes and Schema

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2179: --- Summary: Public API for DataTypes and Schema Key: SPARK-2179 URL: https://issues.apache.org/jira/browse/SPARK-2179 Project: Spark Issue Type: Improveme

[jira] [Created] (SPARK-2178) createSchemaRDD is not thread safe

2014-06-18 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-2178: --- Summary: createSchemaRDD is not thread safe Key: SPARK-2178 URL: https://issues.apache.org/jira/browse/SPARK-2178 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2178) createSchemaRDD is not thread safe

2014-06-18 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-2178: Description: This is because implicit type tags are not thread safe. We could fix this wit

[jira] [Commented] (SPARK-2171) Groovy bindings for Spark

2014-06-18 Thread Artur Andrzejak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035174#comment-14035174 ] Artur Andrzejak commented on SPARK-2171: Hi Patrick, yes, the cool thing is that

[jira] [Closed] (SPARK-433) Adding sorting to RDDs

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-433. - Assignee: Reynold Xin > Adding sorting to RDDs > -- > > Key: SPARK-433

[jira] [Resolved] (SPARK-1072) Use binary search for RangePartitioner when there is more than 1000 partitions

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-1072. Resolution: Fixed Fix Version/s: 0.9.0 Assignee: Holden Karau > Use binary search f

[jira] [Resolved] (SPARK-1933) FileNotFoundException when a directory is passed to SparkContext.addJar/addFile

2014-06-18 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-1933. Resolution: Fixed Fix Version/s: 1.1.0 > FileNotFoundException when a directory is passed to

[jira] [Created] (SPARK-2177) describe table result contains only one column

2014-06-18 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-2177: -- Summary: describe table result contains only one column Key: SPARK-2177 URL: https://issues.apache.org/jira/browse/SPARK-2177 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2176) extra unnecessary exchange operator in group by

2014-06-18 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-2176: -- Summary: extra unnecessary exchange operator in group by Key: SPARK-2176 URL: https://issues.apache.org/jira/browse/SPARK-2176 Project: Spark Issue Type: Bug