[jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525144#comment-15525144 ] Gang Wu commented on SPARK-17672: - They are similar but different. This JIRA deals with the approach to

[jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525137#comment-15525137 ] Sean Owen commented on SPARK-17672: --- I don't see how this is separate from SPARK-17671? > Spark 2.0

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525101#comment-15525101 ] Russell Spitzer edited comment on SPARK-17673 at 9/27/16 5:17 AM: --

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525101#comment-15525101 ] Russell Spitzer commented on SPARK-17673: - {code}== Parsed Logical Plan == Union :- Aggregate

[jira] [Updated] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-09-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-17680: Description: Spark SQL supports Unicode characters for column names when specified within backticks(`).

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525096#comment-15525096 ] Russell Spitzer commented on SPARK-17673: - Ah yeah there would definitely be different pruning in

[jira] [Assigned] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17680: Assignee: Apache Spark > Unicode Character Support for Column Names and Comments >

[jira] [Assigned] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17680: Assignee: (was: Apache Spark) > Unicode Character Support for Column Names and

[jira] [Commented] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525090#comment-15525090 ] Apache Spark commented on SPARK-17680: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-09-26 Thread Xiao Li (JIRA)
Xiao Li created SPARK-17680: --- Summary: Unicode Character Support for Column Names and Comments Key: SPARK-17680 URL: https://issues.apache.org/jira/browse/SPARK-17680 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-17679) Remove unnecessary Py4J ListConverter patch

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17679: Assignee: (was: Apache Spark) > Remove unnecessary Py4J ListConverter patch >

[jira] [Commented] (SPARK-17679) Remove unnecessary Py4J ListConverter patch

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525059#comment-15525059 ] Apache Spark commented on SPARK-17679: -- User 'JasonMWhite' has created a pull request for this

[jira] [Assigned] (SPARK-17679) Remove unnecessary Py4J ListConverter patch

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17679: Assignee: Apache Spark > Remove unnecessary Py4J ListConverter patch >

[jira] [Created] (SPARK-17679) Remove unnecessary Py4J ListConverter patch

2016-09-26 Thread Jason White (JIRA)
Jason White created SPARK-17679: --- Summary: Remove unnecessary Py4J ListConverter patch Key: SPARK-17679 URL: https://issues.apache.org/jira/browse/SPARK-17679 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525036#comment-15525036 ] Herman van Hovell commented on SPARK-17673: --- Could you also share the optimized plan

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525036#comment-15525036 ] Herman van Hovell edited comment on SPARK-17673 at 9/27/16 4:31 AM:

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525011#comment-15525011 ] Reynold Xin commented on SPARK-17673: - The only thing differentiating the two sides of the plan is

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525000#comment-15525000 ] Reynold Xin edited comment on SPARK-17673 at 9/27/16 4:14 AM: --

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525007#comment-15525007 ] Russell Spitzer edited comment on SPARK-17673 at 9/27/16 4:12 AM: --

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525007#comment-15525007 ] Russell Spitzer commented on SPARK-17673: - Looking at this plan ``` Union :-

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525000#comment-15525000 ] Reynold Xin commented on SPARK-17673: - RowDataSourceScanExec.sameResult is probably the problem:

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524999#comment-15524999 ] Russell Spitzer commented on SPARK-17673: - We shouldn't be ... The only thing we cache are

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524993#comment-15524993 ] Herman van Hovell commented on SPARK-17673: --- [~rspitzer] we only reuse an exchange when they

[jira] [Assigned] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17678: Assignee: Apache Spark > Spark 1.6 Scala-2.11 repl doesn't honor

[jira] [Assigned] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17678: Assignee: (was: Apache Spark) > Spark 1.6 Scala-2.11 repl doesn't honor

[jira] [Commented] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524976#comment-15524976 ] Apache Spark commented on SPARK-17678: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Created] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

2016-09-26 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-17678: --- Summary: Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port" Key: SPARK-17678 URL: https://issues.apache.org/jira/browse/SPARK-17678 Project: Spark

[jira] [Commented] (SPARK-17677) Break WindowExec.scala into multiple files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524955#comment-15524955 ] Apache Spark commented on SPARK-17677: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17677) Break WindowExec.scala into multiple files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17677: Assignee: Apache Spark (was: Reynold Xin) > Break WindowExec.scala into multiple files >

[jira] [Assigned] (SPARK-17677) Break WindowExec.scala into multiple files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17677: Assignee: Reynold Xin (was: Apache Spark) > Break WindowExec.scala into multiple files >

[jira] [Created] (SPARK-17677) Break WindowExec.scala into multiple files

2016-09-26 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17677: --- Summary: Break WindowExec.scala into multiple files Key: SPARK-17677 URL: https://issues.apache.org/jira/browse/SPARK-17677 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17676) FsHistoryProvider should ignore hidden files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524893#comment-15524893 ] Apache Spark commented on SPARK-17676: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17676) FsHistoryProvider should ignore hidden files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17676: Assignee: Imran Rashid (was: Apache Spark) > FsHistoryProvider should ignore hidden

[jira] [Assigned] (SPARK-17676) FsHistoryProvider should ignore hidden files

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17676: Assignee: Apache Spark (was: Imran Rashid) > FsHistoryProvider should ignore hidden

[jira] [Created] (SPARK-17676) FsHistoryProvider should ignore hidden files

2016-09-26 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-17676: Summary: FsHistoryProvider should ignore hidden files Key: SPARK-17676 URL: https://issues.apache.org/jira/browse/SPARK-17676 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524848#comment-15524848 ] Russell Spitzer commented on SPARK-17673: - I couldn't get this to happen without C*, hopefully

[jira] [Comment Edited] (SPARK-8425) Add blacklist mechanism for task scheduling

2016-09-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524811#comment-15524811 ] Imran Rashid edited comment on SPARK-8425 at 9/27/16 2:20 AM: -- Breaking off a

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

2016-09-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524811#comment-15524811 ] Imran Rashid commented on SPARK-8425: - Breaking off a smaller chunk of this that can be added

[jira] [Commented] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524790#comment-15524790 ] Apache Spark commented on SPARK-17675: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17675: Assignee: Imran Rashid (was: Apache Spark) > Add Blacklisting of Executors & Nodes

[jira] [Assigned] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17675: Assignee: Apache Spark (was: Imran Rashid) > Add Blacklisting of Executors & Nodes

[jira] [Created] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

2016-09-26 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-17675: Summary: Add Blacklisting of Executors & Nodes within one TaskSet Key: SPARK-17675 URL: https://issues.apache.org/jira/browse/SPARK-17675 Project: Spark

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524734#comment-15524734 ] Russell Spitzer edited comment on SPARK-17673 at 9/27/16 1:39 AM: -- Well

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524734#comment-15524734 ] Russell Spitzer edited comment on SPARK-17673 at 9/27/16 1:38 AM: -- Well

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524734#comment-15524734 ] Russell Spitzer commented on SPARK-17673: - Well in this case they are equal correct? > Reused

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524725#comment-15524725 ] Reynold Xin commented on SPARK-17673: - It's possible if hashCode and equals are not defined properly

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524722#comment-15524722 ] Russell Spitzer commented on SPARK-17673: - Ugh I made a typo in my Parquet Example I don't see it

[jira] [Updated] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17673: Priority: Critical (was: Major) > Reused Exchange Aggregations Produce Incorrect Results >

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524687#comment-15524687 ] Reynold Xin commented on SPARK-17673: - Can you help create a repro (without the need to connect

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524641#comment-15524641 ] Russell Spitzer commented on SPARK-17673: - I only ran this on 2.0.0 and 2.0.1 > Reused Exchange

[jira] [Updated] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

2016-09-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17674: - Description: For example, _currently_ we are having warnings as below:

[jira] [Created] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

2016-09-26 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-17674: Summary: Warnings from SparkR tests being ignored without redirecting to errors Key: SPARK-17674 URL: https://issues.apache.org/jira/browse/SPARK-17674 Project:

[jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524593#comment-15524593 ] Gang Wu commented on SPARK-17672: - Hi [~ajbozarth], can you take a look at the PR? Thanks! > Spark 2.0

[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524595#comment-15524595 ] Gang Wu commented on SPARK-17671: - Hi [~ajbozarth], can you take a look at the PR? Thanks! > Spark 2.0

[jira] [Assigned] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17671: Assignee: (was: Apache Spark) > Spark 2.0 history server summary page is slow even

[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524587#comment-15524587 ] Apache Spark commented on SPARK-17671: -- User 'wgtmac' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17671: Assignee: Apache Spark > Spark 2.0 history server summary page is slow even set >

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524581#comment-15524581 ] Herman van Hovell edited comment on SPARK-17673 at 9/27/16 12:15 AM: -

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524581#comment-15524581 ] Herman van Hovell commented on SPARK-17673: --- [~russell spitzer] Are you using Spark 2.0 or the

[jira] [Comment Edited] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524581#comment-15524581 ] Herman van Hovell edited comment on SPARK-17673 at 9/27/16 12:16 AM: -

[jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524524#comment-15524524 ] Apache Spark commented on SPARK-17672: -- User 'wgtmac' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17672: Assignee: (was: Apache Spark) > Spark 2.0 history server web Ui takes too long for a

[jira] [Assigned] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17672: Assignee: Apache Spark > Spark 2.0 history server web Ui takes too long for a single

[jira] [Updated] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Spitzer updated SPARK-17673: Labels: correctness (was: ) > Reused Exchange Aggregations Produce Incorrect Results >

[jira] [Created] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-26 Thread Russell Spitzer (JIRA)
Russell Spitzer created SPARK-17673: --- Summary: Reused Exchange Aggregations Produce Incorrect Results Key: SPARK-17673 URL: https://issues.apache.org/jira/browse/SPARK-17673 Project: Spark

[jira] [Updated] (SPARK-6624) Convert filters into CNF for data sources

2016-09-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-6624: --- Assignee: (was: Yijie Shen) > Convert filters into CNF for data sources >

[jira] [Commented] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524424#comment-15524424 ] Gang Wu commented on SPARK-17672: - I'm working on a fix and will send a PR soon. > Spark 2.0 history

[jira] [Updated] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Wu updated SPARK-17672: Description: When there are 10K application history in the history server back end, it can take a very

[jira] [Created] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-26 Thread Gang Wu (JIRA)
Gang Wu created SPARK-17672: --- Summary: Spark 2.0 history server web Ui takes too long for a single application Key: SPARK-17672 URL: https://issues.apache.org/jira/browse/SPARK-17672 Project: Spark

[jira] [Commented] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524175#comment-15524175 ] Xiao Li commented on SPARK-17653: - Since Simon already submitted the PR, I will not continue the

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Target Version/s: 2.0.2, 2.1.0 Priority: Critical (was: Major) > take() or isEmpty() on

[jira] [Created] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Gang Wu (JIRA)
Gang Wu created SPARK-17671: --- Summary: Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications Key: SPARK-17671 URL: https://issues.apache.org/jira/browse/SPARK-17671

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Component/s: (was: Java API) SQL > take() or isEmpty() on dataset leaks s3a

[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-26 Thread Gang Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524173#comment-15524173 ] Gang Wu commented on SPARK-17671: - I'm working on this and will send a pull request soon. > Spark 2.0

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

2016-09-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524166#comment-15524166 ] Sean Owen commented on SPARK-17669: --- It's hard to say what is going on without knowing what you're

[jira] [Assigned] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17666: Assignee: Apache Spark > take() or isEmpty() on dataset leaks s3a connections >

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524108#comment-15524108 ] Apache Spark commented on SPARK-17666: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17666: Assignee: (was: Apache Spark) > take() or isEmpty() on dataset leaks s3a connections

[jira] [Created] (SPARK-17670) Spark DataFrame/Dataset no longer supports Option[Map] in case classes

2016-09-26 Thread Daniel Williams (JIRA)
Daniel Williams created SPARK-17670: --- Summary: Spark DataFrame/Dataset no longer supports Option[Map] in case classes Key: SPARK-17670 URL: https://issues.apache.org/jira/browse/SPARK-17670

[jira] [Updated] (SPARK-17669) Strange behavior using Datasets

2016-09-26 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miles Crawford updated SPARK-17669: --- Summary: Strange behavior using Datasets (was: Strange UI behavior using Datasets) >

[jira] [Created] (SPARK-17669) Strange UI behavior using Datasets

2016-09-26 Thread Miles Crawford (JIRA)
Miles Crawford created SPARK-17669: -- Summary: Strange UI behavior using Datasets Key: SPARK-17669 URL: https://issues.apache.org/jira/browse/SPARK-17669 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8824) Support Parquet time related logical types

2016-09-26 Thread Nate Sammons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524073#comment-15524073 ] Nate Sammons commented on SPARK-8824: - Any progress on these items? Specifically TIMESTAMP_MILLIS for

[jira] [Updated] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-26 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-17652: - Assignee: Sameer Agarwal > Fix confusing exception message while reserving capacity >

[jira] [Resolved] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-26 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-17652. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 Issue resolved by pull request

[jira] [Resolved] (SPARK-17153) [Structured streams] readStream ignores partition columns

2016-09-26 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17153. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14803

[jira] [Updated] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

2016-09-26 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-17668: -- Summary: Support representing structs with case classes and tuples in spark sql udf inputs

[jira] [Commented] (SPARK-17668) Support case classes and tuples to represent structs in spark sql udfs

2016-09-26 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523989#comment-15523989 ] koert kuipers commented on SPARK-17668: --- original conversation is here:

[jira] [Created] (SPARK-17668) Support case classes and tuples to represent structs in spark sql udfs

2016-09-26 Thread koert kuipers (JIRA)
koert kuipers created SPARK-17668: - Summary: Support case classes and tuples to represent structs in spark sql udfs Key: SPARK-17668 URL: https://issues.apache.org/jira/browse/SPARK-17668 Project:

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523967#comment-15523967 ] Sean Owen commented on SPARK-17666: --- Agree, it might happen to help some cases or even this one but

[jira] [Created] (SPARK-17667) Make locking fine grained in YarnAllocator#enqueueGetLossReasonRequest

2016-09-26 Thread Ashwin Shankar (JIRA)
Ashwin Shankar created SPARK-17667: -- Summary: Make locking fine grained in YarnAllocator#enqueueGetLossReasonRequest Key: SPARK-17667 URL: https://issues.apache.org/jira/browse/SPARK-17667 Project:

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523921#comment-15523921 ] Josh Rosen commented on SPARK-17666: I think that one problem with that approach is that any

[jira] [Commented] (SPARK-17665) SparkR does not support options in other types consistently other APIs

2016-09-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523844#comment-15523844 ] Felix Cheung commented on SPARK-17665: -- supporting just character and logical seem fine. AFAIK we

[jira] [Updated] (SPARK-17638) Stop JVM StreamingContext when the Python process is dead

2016-09-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17638: - Fix Version/s: (was: 2.0.2) 2.0.1 > Stop JVM StreamingContext when the

[jira] [Comment Edited] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523799#comment-15523799 ] Sean Owen edited comment on SPARK-17666 at 9/26/16 6:24 PM: Hm, I wonder if a

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523799#comment-15523799 ] Sean Owen commented on SPARK-17666: --- Hm, I wonder if a couple problems of this form could be solved by

[jira] [Updated] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17649: - Fix Version/s: 1.6.3 > Log how many Spark events got dropped in LiveListenerBus >

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523742#comment-15523742 ] Josh Rosen commented on SPARK-17666: My hunch is that there's cleanup which is performed in a

[jira] [Resolved] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17649. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 > Log how many Spark

[jira] [Updated] (SPARK-17454) Use Mesos disk resources

2016-09-26 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-17454: Summary: Use Mesos disk resources (was: Add option to specify Mesos resource offer

[jira] [Comment Edited] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2016-09-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523623#comment-15523623 ] Xiangrui Meng edited comment on SPARK-17647 at 9/26/16 5:07 PM: Thanks

  1   2   >