[jira] [Commented] (SPARK-12519) "Managed memory leak detected" when using distinct on PySpark DataFrame

2016-02-05 Thread Jason C Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135630#comment-15135630 ] Jason C Lee commented on SPARK-12519: - The memory leak in question comes from a Tungs

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Description: The CheckpointInterval of DStream could be customized to multiple of BatchInteveal. So when

[jira] [Assigned] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13222: Assignee: (was: Apache Spark) > checkpoint latest stateful RDD on graceful shutdown >

[jira] [Commented] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135611#comment-15135611 ] Apache Spark commented on SPARK-13222: -- User 'mwws' has created a pull request for t

[jira] [Assigned] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13222: Assignee: Apache Spark > checkpoint latest stateful RDD on graceful shutdown > ---

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Description: In case the input source is not repeatable, and user don't want to enable WAL because of the

[jira] [Updated] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mao, Wei updated SPARK-13222: - Component/s: Streaming > checkpoint latest stateful RDD on graceful shutdown > --

[jira] [Created] (SPARK-13222) checkpoint latest stateful RDD on graceful shutdown

2016-02-05 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-13222: Summary: checkpoint latest stateful RDD on graceful shutdown Key: SPARK-13222 URL: https://issues.apache.org/jira/browse/SPARK-13222 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-13171) Update promise & future to Promise and Future as the old ones are deprecated

2016-02-05 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13171. - Resolution: Fixed Assignee: Jakob Odersky Fix Version/s: 2.0.0 > Update promise &

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2016-02-05 Thread Cristian Opris (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135447#comment-15135447 ] Cristian Opris commented on SPARK-11725: You're right, sorry it seems I lost trac

[jira] [Assigned] (SPARK-13221) GroupingSets Returns an Incorrect Results

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13221: Assignee: Apache Spark > GroupingSets Returns an Incorrect Results > -

[jira] [Assigned] (SPARK-13221) GroupingSets Returns an Incorrect Results

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13221: Assignee: (was: Apache Spark) > GroupingSets Returns an Incorrect Results > --

[jira] [Commented] (SPARK-13221) GroupingSets Returns an Incorrect Results

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135446#comment-15135446 ] Apache Spark commented on SPARK-13221: -- User 'gatorsmile' has created a pull request

[jira] [Created] (SPARK-13221) GroupingSets Returns an Incorrect Results

2016-02-05 Thread Xiao Li (JIRA)
Xiao Li created SPARK-13221: --- Summary: GroupingSets Returns an Incorrect Results Key: SPARK-13221 URL: https://issues.apache.org/jira/browse/SPARK-13221 Project: Spark Issue Type: Bug Com

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135386#comment-15135386 ] Xiao Li commented on SPARK-12720: - The solution is done, but hitting a bug in grouping se

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2016-02-05 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135357#comment-15135357 ] Jeff Zhang commented on SPARK-11725: +1 for Option[T] > Let UDF to handle null value

[jira] [Assigned] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13068: Assignee: (was: Apache Spark) > Extend pyspark ml paramtype conversion to support list

[jira] [Assigned] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13068: Assignee: Apache Spark > Extend pyspark ml paramtype conversion to support lists > ---

[jira] [Commented] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135330#comment-15135330 ] Apache Spark commented on SPARK-13068: -- User 'sethah' has created a pull request for

[jira] [Created] (SPARK-13220) Deprecate "yarn-client" and "yarn-cluster"

2016-02-05 Thread Andrew Or (JIRA)
Andrew Or created SPARK-13220: - Summary: Deprecate "yarn-client" and "yarn-cluster" Key: SPARK-13220 URL: https://issues.apache.org/jira/browse/SPARK-13220 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2016-02-05 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135264#comment-15135264 ] Michael Armbrust commented on SPARK-11725: -- [~onetoinfin...@yahoo.com], unfortun

[jira] [Commented] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Jakob Odersky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135254#comment-15135254 ] Jakob Odersky commented on SPARK-13176: --- The PR I submitted uses java.nio.Files, it

[jira] [Assigned] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13176: Assignee: Apache Spark > Ignore deprecation warning for ProcessBuilder lines_! > -

[jira] [Commented] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135251#comment-15135251 ] Apache Spark commented on SPARK-13176: -- User 'jodersky' has created a pull request f

[jira] [Assigned] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13176: Assignee: (was: Apache Spark) > Ignore deprecation warning for ProcessBuilder lines_!

[jira] [Commented] (SPARK-13216) Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

2016-02-05 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135233#comment-15135233 ] Hari Shreedharan commented on SPARK-13216: -- I think this brings up a broader que

[jira] [Commented] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Jakob Odersky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135222#comment-15135222 ] Jakob Odersky commented on SPARK-13176: --- One of the places the process api is used

[jira] [Updated] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-05 Thread Abhinav Chawade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Chawade updated SPARK-13219: Description: When 2 or more tables are joined in SparkSQL and there is an equality clause i

[jira] [Updated] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-05 Thread Abhinav Chawade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Chawade updated SPARK-13219: Description: When 2 or more tables are joined in SparkSQL and there is an equality clause i

[jira] [Resolved] (SPARK-13215) Remove fallback in codegen

2016-02-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-13215. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11097 [https://github.

[jira] [Updated] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-05 Thread Abhinav Chawade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Chawade updated SPARK-13219: Description: When 2 or more tables are joined in SparkSQL and there is an equality clause i

[jira] [Created] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-05 Thread Abhinav Chawade (JIRA)
Abhinav Chawade created SPARK-13219: --- Summary: Pushdown predicate propagation in SparkSQL with join Key: SPARK-13219 URL: https://issues.apache.org/jira/browse/SPARK-13219 Project: Spark Is

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135206#comment-15135206 ] Xiao Li commented on SPARK-12720: - Ok, I think we should not use subquery to do it. > SQ

[jira] [Issue Comment Deleted] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12720: Comment: was deleted (was: I guess the best way is to add another `Project` above `Aggregate`) > SQL gener

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135187#comment-15135187 ] Xiao Li commented on SPARK-12720: - I guess the best way is to add another `Project` above

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135177#comment-15135177 ] Xiao Li commented on SPARK-12720: - {code}SELECT key, value, count(value) FROM t1 GROUP BY

[jira] [Resolved] (SPARK-13002) Mesos scheduler backend does not follow the property spark.dynamicAllocation.initialExecutors

2016-02-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13002. --- Resolution: Fixed Assignee: Luc Bourlier Fix Version/s: 2.0.0 > Mesos scheduler backe

[jira] [Updated] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13214: -- Assignee: Bill Chambers > Fix dynamic allocation docs > --- > >

[jira] [Resolved] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13214. --- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 > Fix dynamic allocation docs

[jira] [Updated] (SPARK-12939) migrate encoder resolution to Analyzer

2016-02-05 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-12939: - Assignee: Wenchen Fan > migrate encoder resolution to Analyzer >

[jira] [Resolved] (SPARK-12939) migrate encoder resolution to Analyzer

2016-02-05 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-12939. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10852 [htt

[jira] [Commented] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135163#comment-15135163 ] Sean Owen commented on SPARK-13202: --- Actually, try spark-submit. This is a class loader

[jira] [Commented] (SPARK-13218) Executor failed after SparkContext start and start again

2016-02-05 Thread leo wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135153#comment-15135153 ] leo wu commented on SPARK-13218: One more point : If spark-default.conf and spark-env.

[jira] [Created] (SPARK-13218) Executor failed after SparkContext start and start again

2016-02-05 Thread leo wu (JIRA)
leo wu created SPARK-13218: -- Summary: Executor failed after SparkContext start and start again Key: SPARK-13218 URL: https://issues.apache.org/jira/browse/SPARK-13218 Project: Spark Issue Type: Bu

[jira] [Closed] (SPARK-13217) UI of visualization of stage

2016-02-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-13217. -- Resolution: Not A Problem It works after restart the driver. > UI of visualization of stage > ---

[jira] [Created] (SPARK-13217) UI of visualization of stage

2016-02-05 Thread Davies Liu (JIRA)
Davies Liu created SPARK-13217: -- Summary: UI of visualization of stage Key: SPARK-13217 URL: https://issues.apache.org/jira/browse/SPARK-13217 Project: Spark Issue Type: Improvement C

[jira] [Commented] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.

2016-02-05 Thread leo wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135036#comment-15135036 ] leo wu commented on SPARK-13198: Hi, Sean I am trying to do this similar in an IPyt

[jira] [Assigned] (SPARK-13215) Remove fallback in codegen

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13215: Assignee: Apache Spark > Remove fallback in codegen > -- > >

[jira] [Commented] (SPARK-13215) Remove fallback in codegen

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135009#comment-15135009 ] Apache Spark commented on SPARK-13215: -- User 'davies' has created a pull request for

[jira] [Assigned] (SPARK-13215) Remove fallback in codegen

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13215: Assignee: (was: Apache Spark) > Remove fallback in codegen > -

[jira] [Created] (SPARK-13216) Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

2016-02-05 Thread Neelesh Srinivas Salian (JIRA)
Neelesh Srinivas Salian created SPARK-13216: --- Summary: Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint Key: SPARK-13216 URL: https://issues.apac

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134988#comment-15134988 ] Asif Hussain Shahid commented on SPARK-13116: - It will show up if you throw a

[jira] [Created] (SPARK-13215) Remove fallback in codegen

2016-02-05 Thread Davies Liu (JIRA)
Davies Liu created SPARK-13215: -- Summary: Remove fallback in codegen Key: SPARK-13215 URL: https://issues.apache.org/jira/browse/SPARK-13215 Project: Spark Issue Type: Improvement Re

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134979#comment-15134979 ] Davies Liu commented on SPARK-13116: I tried the test you attached, it work well on m

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134969#comment-15134969 ] Asif Hussain Shahid commented on SPARK-13116: - Now the unary expression was o

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134968#comment-15134968 ] Asif Hussain Shahid commented on SPARK-13116: - No, the unary expression was o

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134959#comment-15134959 ] Davies Liu commented on SPARK-13116: Which UnaryExpression generate incorrect code? I

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134929#comment-15134929 ] Asif Hussain Shahid commented on SPARK-13116: - I further debugged the issue

[jira] [Updated] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-05 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif Hussain Shahid updated SPARK-13116: Attachment: SPARK_13116_Test.scala A test which reproduces the issue. Pls note that

[jira] [Commented] (SPARK-13166) Remove DataStreamReader/Writer

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134890#comment-15134890 ] Apache Spark commented on SPARK-13166: -- User 'zsxwing' has created a pull request fo

[jira] [Commented] (SPARK-12430) Temporary folders do not get deleted after Task completes causing problems with disk space.

2016-02-05 Thread Fede Bar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134874#comment-15134874 ] Fede Bar commented on SPARK-12430: -- Very well explained thanks. I'll try the new build a

[jira] [Commented] (SPARK-13210) NPE in Sort

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134828#comment-15134828 ] Apache Spark commented on SPARK-13210: -- User 'davies' has created a pull request for

[jira] [Commented] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134777#comment-15134777 ] Apache Spark commented on SPARK-13214: -- User 'anabranch' has created a pull request

[jira] [Updated] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13214: -- Affects Version/s: 1.6.0 Target Version/s: 1.6.1, 2.0.0 Component/s: Documentation > Fix

[jira] [Commented] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134746#comment-15134746 ] Apache Spark commented on SPARK-13214: -- User 'anabranch' has created a pull request

[jira] [Assigned] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13214: Assignee: Apache Spark > Fix dynamic allocation docs > --- > >

[jira] [Assigned] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13214: Assignee: (was: Apache Spark) > Fix dynamic allocation docs >

[jira] [Created] (SPARK-13214) Update docs to reflect dynamicAllocation to be true

2016-02-05 Thread Bill Chambers (JIRA)
Bill Chambers created SPARK-13214: - Summary: Update docs to reflect dynamicAllocation to be true Key: SPARK-13214 URL: https://issues.apache.org/jira/browse/SPARK-13214 Project: Spark Issue T

[jira] [Updated] (SPARK-13214) Fix dynamic allocation docs

2016-02-05 Thread Bill Chambers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Chambers updated SPARK-13214: -- Description: Update docs to reflect dynamicAllocation to be available for all cluster managers

[jira] [Created] (SPARK-13213) BroadcastNestedLoopJoin is very slow

2016-02-05 Thread Davies Liu (JIRA)
Davies Liu created SPARK-13213: -- Summary: BroadcastNestedLoopJoin is very slow Key: SPARK-13213 URL: https://issues.apache.org/jira/browse/SPARK-13213 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-13180) Protect against SessionState being null when accessing HiveClientImpl#conf

2016-02-05 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated SPARK-13180: --- Attachment: spark-13180-util.patch Patch with HiveConfUtil.scala For record. > Protect against SessionState

[jira] [Commented] (SPARK-13189) Cleanup build references to Scala 2.10

2016-02-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134632#comment-15134632 ] Apache Spark commented on SPARK-13189: -- User 'lresende' has created a pull request f

[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

2016-02-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134414#comment-15134414 ] Steve Loughran commented on SPARK-12807: mixing dependency versions in a single p

[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

2016-02-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134412#comment-15134412 ] Steve Loughran commented on SPARK-12807: It's good to hear that things work with

[jira] [Comment Edited] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-05 Thread Michael Schmitz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134332#comment-15134332 ] Michael Schmitz edited comment on SPARK-13202 at 2/5/16 3:40 PM: --

[jira] [Commented] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-05 Thread Michael Schmitz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134332#comment-15134332 ] Michael Schmitz commented on SPARK-13202: - [~srowen] in that case the documentati

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2016-02-05 Thread Cristian Opris (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134322#comment-15134322 ] Cristian Opris commented on SPARK-11725: I would argue this is a rather inappropr

[jira] [Updated] (SPARK-12969) Exception while casting a spark supported date formatted "string" to "date" data type.

2016-02-05 Thread Jais Sebastian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jais Sebastian updated SPARK-12969: --- Description: Getting exception while converting a string column( column is having spark sup

[jira] [Commented] (SPARK-12969) Exception while casting a spark supported date formatted "string" to "date" data type.

2016-02-05 Thread Jais Sebastian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134287#comment-15134287 ] Jais Sebastian commented on SPARK-12969: Hi , Looks like this issue happens only

[jira] [Created] (SPARK-13212) Provide a way to unregister data sources from a SQLContext

2016-02-05 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-13212: -- Summary: Provide a way to unregister data sources from a SQLContext Key: SPARK-13212 URL: https://issues.apache.org/jira/browse/SPARK-13212 Project: Spark

[jira] [Commented] (SPARK-10548) Concurrent execution in SQL does not work

2016-02-05 Thread Akshay Harale (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134146#comment-15134146 ] Akshay Harale commented on SPARK-10548: --- We are still facing this issue while repea

[jira] [Updated] (SPARK-12739) Details of batch in Streaming tab uses two Duration columns

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12739: -- Assignee: Mario Briggs > Details of batch in Streaming tab uses two Duration columns >

[jira] [Commented] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134028#comment-15134028 ] Sean Owen commented on SPARK-13198: --- I don't see evidence of a problem here yet. Stuff

[jira] [Commented] (SPARK-13199) Upgrade apache httpclient version to the latest 4.5 for security

2016-02-05 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134022#comment-15134022 ] Ted Yu commented on SPARK-13199: This task should be done when there is a Hadoop release

[jira] [Commented] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134010#comment-15134010 ] Sean Owen commented on SPARK-13176: --- Can we possibly not use this API at all? Sorry hav

[jira] [Commented] (SPARK-13204) Replace use of mutable.SynchronizedMap with ConcurrentHashMap

2016-02-05 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134006#comment-15134006 ] Ted Yu commented on SPARK-13204: Just noticed that parent JIRA was marked trivial. Shou

[jira] [Commented] (SPARK-13172) Stop using RichException.getStackTrace it is deprecated

2016-02-05 Thread sachin aggarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133963#comment-15133963 ] sachin aggarwal commented on SPARK-13172: - as scala also recommends the same htt

[jira] [Resolved] (SPARK-13188) There is OMM when a receiver stores block

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13188. --- Resolution: Not A Problem Fix Version/s: (was: 2.0.0) Target Version/s: (was: 2

[jira] [Reopened] (SPARK-13039) Spark Streaming with Mesos shutdown without any reason on logs

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-13039: --- > Spark Streaming with Mesos shutdown without any reason on logs > --

[jira] [Resolved] (SPARK-13039) Spark Streaming with Mesos shutdown without any reason on logs

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13039. --- Resolution: Not A Problem > Spark Streaming with Mesos shutdown without any reason on logs >

[jira] [Resolved] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13202. --- Resolution: Not A Problem You have a case where you need your classes available at the same level as

[jira] [Comment Edited] (SPARK-13177) Update ActorWordCount example to not directly use low level linked list as it is deprecated.

2016-02-05 Thread sachin aggarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133954#comment-15133954 ] sachin aggarwal edited comment on SPARK-13177 at 2/5/16 10:23 AM: -

[jira] [Commented] (SPARK-13177) Update ActorWordCount example to not directly use low level linked list as it is deprecated.

2016-02-05 Thread sachin aggarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133954#comment-15133954 ] sachin aggarwal commented on SPARK-13177: - Hi [~AlHolden], I will like to work o

[jira] [Resolved] (SPARK-13192) Some memory leak problem?

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13192. --- Resolution: Invalid Questions go to user@ > Some memory leak problem? > - >

[jira] [Commented] (SPARK-13191) Update LICENSE with Scala 2.11 dependencies

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133952#comment-15133952 ] Sean Owen commented on SPARK-13191: --- It is really not necessary to make so many PRs and

[jira] [Updated] (SPARK-13199) Upgrade apache httpclient version to the latest 4.5 for security

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-13199: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) This also was not filled.out co

[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133946#comment-15133946 ] Sean Owen commented on SPARK-12807: --- Why vary this to use an earlier version - seems li

[jira] [Resolved] (SPARK-13204) Replace use of mutable.SynchronizedMap with ConcurrentHashMap

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13204. --- Resolution: Duplicate > Replace use of mutable.SynchronizedMap with ConcurrentHashMap > -

[jira] [Updated] (SPARK-13204) Replace use of mutable.SynchronizedMap with ConcurrentHashMap

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-13204: -- Priority: Trivial (was: Major) Component/s: Spark Core Issue Type: Improvement (was: Bug)

[jira] [Commented] (SPARK-13204) Replace use of mutable.SynchronizedMap with ConcurrentHashMap

2016-02-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133940#comment-15133940 ] Sean Owen commented on SPARK-13204: --- I think you have been around long enough to know t

  1   2   >