[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438983#comment-16438983 ] Xiao Li commented on SPARK-23206: - No permission to access the doc 

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438982#comment-16438982 ] Wenchen Fan commented on SPARK-23715: - is there a standard definition of from_utc_timestamp? How

[jira] [Resolved] (SPARK-23917) High-order function: array_max(x) → x

2018-04-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23917. - Resolution: Fixed Fix Version/s: 2.4.0 > High-order function: array_max(x) → x >

[jira] [Assigned] (SPARK-23917) High-order function: array_max(x) → x

2018-04-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23917: --- Assignee: Marco Gaido > High-order function: array_max(x) → x >

[jira] [Resolved] (SPARK-23956) Use effective RPC port in AM registration

2018-04-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23956. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21047

[jira] [Assigned] (SPARK-23956) Use effective RPC port in AM registration

2018-04-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23956: --- Assignee: Gera Shegalov > Use effective RPC port in AM registration >

[jira] [Comment Edited] (SPARK-23929) pandas_udf schema mapped by position and not by name

2018-04-15 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438935#comment-16438935 ] Li Jin edited comment on SPARK-23929 at 4/16/18 2:40 AM: - I agree with

[jira] [Commented] (SPARK-23929) pandas_udf schema mapped by position and not by name

2018-04-15 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438935#comment-16438935 ] Li Jin commented on SPARK-23929: I agree with [~hyukjin.kwon]. Seems like there is not a strong enough

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-15 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: SPARK-23206 Design Doc.pdf) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-15 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: SPARK-23206 Design Doc.pdf > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-19618) Inconsistency wrt max. buckets allowed from Dataframe API vs SQL

2018-04-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438912#comment-16438912 ] Wenchen Fan commented on SPARK-19618: - making it configurable sounds like a good idea, can you open a

[jira] [Commented] (SPARK-23982) NoSuchMethodException: There is no startCredentialUpdater method in the object YarnSparkHadoopUtil

2018-04-15 Thread John (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438895#comment-16438895 ] John commented on SPARK-23982: -- see spark-core_2.11-2.3.0 and spark-yarn_2.11-2.3.0

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438885#comment-16438885 ] Kazuaki Ishizaki commented on SPARK-23986: -- While I also checked it with branch-2.3, it works

[jira] [Commented] (SPARK-23983) Disable X-Frame-Options from Spark UI response headers if explicitly configured

2018-04-15 Thread Joe Pallas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438880#comment-16438880 ] Joe Pallas commented on SPARK-23983: I'm just a bystander.  I found a couple of bug reports that

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-15 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: SPARK-23206 Design Doc.pdf > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-15 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu commented on SPARK-23206: --- After discussion with [~irashid] on the PR, we've decided to move

[jira] [Assigned] (SPARK-23988) [Mesos] Improve handling of appResource in mesos dispatcher when using Docker

2018-04-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23988: Assignee: (was: Apache Spark) > [Mesos] Improve handling of appResource in mesos

[jira] [Commented] (SPARK-23988) [Mesos] Improve handling of appResource in mesos dispatcher when using Docker

2018-04-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438875#comment-16438875 ] Apache Spark commented on SPARK-23988: -- User 'pmackles' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23988) [Mesos] Improve handling of appResource in mesos dispatcher when using Docker

2018-04-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23988: Assignee: Apache Spark > [Mesos] Improve handling of appResource in mesos dispatcher when

[jira] [Created] (SPARK-23988) [Mesos] Improve handling of appResource in mesos dispatcher when using Docker

2018-04-15 Thread paul mackles (JIRA)
paul mackles created SPARK-23988: Summary: [Mesos] Improve handling of appResource in mesos dispatcher when using Docker Key: SPARK-23988 URL: https://issues.apache.org/jira/browse/SPARK-23988

[jira] [Commented] (SPARK-19618) Inconsistency wrt max. buckets allowed from Dataframe API vs SQL

2018-04-15 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438822#comment-16438822 ] Fernando Pereira commented on SPARK-19618: -- Is there any technological problem in using more

[jira] [Created] (SPARK-23987) Unused mailing lists

2018-04-15 Thread Sebb (JIRA)
Sebb created SPARK-23987: Summary: Unused mailing lists Key: SPARK-23987 URL: https://issues.apache.org/jira/browse/SPARK-23987 Project: Spark Issue Type: Task Components: Documentation

[jira] [Comment Edited] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438818#comment-16438818 ] Kazuaki Ishizaki edited comment on SPARK-23986 at 4/15/18 7:36 PM: ---

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438818#comment-16438818 ] Kazuaki Ishizaki commented on SPARK-23986: -- Thank for reporting an issue with deep dive. When I

[jira] [Closed] (SPARK-22014) Sample windows in Spark SQL

2018-04-15 Thread Simon Schiff (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Schiff closed SPARK-22014. > Sample windows in Spark SQL > --- > > Key: SPARK-22014 >

[jira] [Resolved] (SPARK-22014) Sample windows in Spark SQL

2018-04-15 Thread Simon Schiff (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Schiff resolved SPARK-22014. -- Resolution: Auto Closed > Sample windows in Spark SQL > --- > >

[jira] [Commented] (SPARK-23983) Disable X-Frame-Options from Spark UI response headers if explicitly configured

2018-04-15 Thread Taylor Cressy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438783#comment-16438783 ] Taylor Cressy commented on SPARK-23983: --- [~jpallas] That is unfortunate. Do we have a support

[jira] [Commented] (SPARK-21811) Inconsistency when finding the widest common type of a combination of DateType, StringType, and NumericType

2018-04-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438737#comment-16438737 ] Apache Spark commented on SPARK-21811: -- User 'jiangxb1987' has created a pull request for this

[jira] [Resolved] (SPARK-23959) UnresolvedException with DataSet created from Seq.empty since Spark 2.3.0

2018-04-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23959. -- Resolution: Cannot Reproduce I can't reproduce in the master too. I am resolving this. It'd be

[jira] [Updated] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-15 Thread Michel Davit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Davit updated SPARK-23986: - Priority: Major (was: Minor) > CompileException when using too many avg aggregation after

[jira] [Created] (SPARK-23985) predicate push down doesn't work with simple compound partition spec

2018-04-15 Thread Ohad Raviv (JIRA)
Ohad Raviv created SPARK-23985: -- Summary: predicate push down doesn't work with simple compound partition spec Key: SPARK-23985 URL: https://issues.apache.org/jira/browse/SPARK-23985 Project: Spark

[jira] [Created] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-15 Thread Michel Davit (JIRA)
Michel Davit created SPARK-23986: Summary: CompileException when using too many avg aggregation after joining Key: SPARK-23986 URL: https://issues.apache.org/jira/browse/SPARK-23986 Project: Spark

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-04-15 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438591#comment-16438591 ] Izek Greenfield commented on SPARK-23904: - [~viirya] this does not help because spark creates