[jira] [Created] (SPARK-24749) Cannot filter array with named_struct

pin_zhang created SPARK-24749: - Summary: Cannot filter array with named_struct Key: SPARK-24749 URL: https://issues.apache.org/jira/browse/SPARK-24749 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24746) AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region.

[ https://issues.apache.org/jira/browse/SPARK-24746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534443#comment-16534443 ] Liang-Chi Hsieh commented on SPARK-24746: - Maybe related:

[jira] [Commented] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534437#comment-16534437 ] Apache Spark commented on SPARK-24742: -- User 'kupferk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24742: Assignee: Apache Spark > Field Metadata raises NullPointerException in hashCode method >

[jira] [Assigned] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24742: Assignee: (was: Apache Spark) > Field Metadata raises NullPointerException in

[jira] [Comment Edited] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533286#comment-16533286 ] Kaya Kupferschmidt edited comment on SPARK-24742 at 7/6/18 5:15 AM:

[jira] [Resolved] (SPARK-24692) Improvement FilterPushdownBenchmark

[ https://issues.apache.org/jira/browse/SPARK-24692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24692. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21677

[jira] [Assigned] (SPARK-24692) Improvement FilterPushdownBenchmark

[ https://issues.apache.org/jira/browse/SPARK-24692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24692: Assignee: Yuming Wang > Improvement FilterPushdownBenchmark >

[jira] [Resolved] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24737. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21713

[jira] [Assigned] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24737: Assignee: Takuya Ueshin > Type coercion between StructTypes. >

[jira] [Commented] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534371#comment-16534371 ] Apache Spark commented on SPARK-24748: -- User 'arunmahadevan' has created a pull request for this

[jira] [Assigned] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24748: Assignee: Apache Spark > Support for reporting custom metrics via Streaming Query

[jira] [Assigned] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

[ https://issues.apache.org/jira/browse/SPARK-24748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24748: Assignee: (was: Apache Spark) > Support for reporting custom metrics via Streaming

[jira] [Created] (SPARK-24748) Support for reporting custom metrics via Streaming Query Progress

Arun Mahadevan created SPARK-24748: -- Summary: Support for reporting custom metrics via Streaming Query Progress Key: SPARK-24748 URL: https://issues.apache.org/jira/browse/SPARK-24748 Project: Spark

[jira] [Commented] (SPARK-24164) Support column list as the pivot column in Pivot

[ https://issues.apache.org/jira/browse/SPARK-24164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534234#comment-16534234 ] Apache Spark commented on SPARK-24164: -- User 'maryannxue' has created a pull request for this

[jira] [Assigned] (SPARK-24164) Support column list as the pivot column in Pivot

[ https://issues.apache.org/jira/browse/SPARK-24164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24164: Assignee: Apache Spark (was: Maryann Xue) > Support column list as the pivot column in

[jira] [Assigned] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

[ https://issues.apache.org/jira/browse/SPARK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24163: Assignee: (was: Apache Spark) > Support "ANY" or sub-query for Pivot "IN" clause >

[jira] [Assigned] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

[ https://issues.apache.org/jira/browse/SPARK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24163: Assignee: Apache Spark > Support "ANY" or sub-query for Pivot "IN" clause >

[jira] [Commented] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

[ https://issues.apache.org/jira/browse/SPARK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534233#comment-16534233 ] Apache Spark commented on SPARK-24163: -- User 'maryannxue' has created a pull request for this

[jira] [Assigned] (SPARK-24164) Support column list as the pivot column in Pivot

[ https://issues.apache.org/jira/browse/SPARK-24164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24164: Assignee: Maryann Xue (was: Apache Spark) > Support column list as the pivot column in

[jira] [Assigned] (SPARK-24694) Integration tests pass only one app argument

[ https://issues.apache.org/jira/browse/SPARK-24694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24694: - Assignee: Stavros Kontopoulos > Integration tests pass only one app argument >

[jira] [Resolved] (SPARK-24694) Integration tests pass only one app argument

[ https://issues.apache.org/jira/browse/SPARK-24694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24694. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21672

[jira] [Updated] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24747: -- Shepherd: Xiangrui Meng > Make spark.ml.util.Instrumentation class more flexible >

[jira] [Assigned] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24747: - Assignee: Bago Amirbekian > Make spark.ml.util.Instrumentation class more flexible >

[jira] [Commented] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533998#comment-16533998 ] Apache Spark commented on SPARK-24747: -- User 'MrBago' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24747: Assignee: (was: Apache Spark) > Make spark.ml.util.Instrumentation class more

[jira] [Assigned] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24747: Assignee: Apache Spark > Make spark.ml.util.Instrumentation class more flexible >

[jira] [Created] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

Bago Amirbekian created SPARK-24747: --- Summary: Make spark.ml.util.Instrumentation class more flexible Key: SPARK-24747 URL: https://issues.apache.org/jira/browse/SPARK-24747 Project: Spark

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533977#comment-16533977 ] Mukul Murthy commented on SPARK-24438: -- Are null and empty string both invalid partition values? I

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533929#comment-16533929 ] Dongjoon Hyun commented on SPARK-24438: --- Yep. It works as designed for now. Shall we close this

[jira] [Resolved] (SPARK-24675) Rename table: validate existence of new location

[ https://issues.apache.org/jira/browse/SPARK-24675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24675. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.4.0 > Rename table:

[jira] [Created] (SPARK-24746) AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region.

Kushagra Singh created SPARK-24746: -- Summary: AWS S3 301 Moved Permanently error message even after setting fs.s3a.endpoint for bucket in Mumbai region. Key: SPARK-24746 URL:

[jira] [Resolved] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-24743. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21717

[jira] [Assigned] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24743: - Assignee: luochuan Priority: Minor (was: Major) > Update the JavaDirectKafkaWordCount

[jira] [Assigned] (SPARK-24711) Integration tests will not work with exclude/include tags

[ https://issues.apache.org/jira/browse/SPARK-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24711: - Assignee: Stavros Kontopoulos > Integration tests will not work with exclude/include tags >

[jira] [Resolved] (SPARK-24711) Integration tests will not work with exclude/include tags

[ https://issues.apache.org/jira/browse/SPARK-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24711. --- Resolution: Fixed Issue resolved by pull request 21697 [https://github.com/apache/spark/pull/21697]

[jira] [Resolved] (SPARK-23820) Allow the long form of call sites to be recorded in the log

[ https://issues.apache.org/jira/browse/SPARK-23820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23820. --- Resolution: Fixed Fix Version/s: 2.4.0 Resolved by 

[jira] [Assigned] (SPARK-23820) Allow the long form of call sites to be recorded in the log

[ https://issues.apache.org/jira/browse/SPARK-23820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23820: - Assignee: Michael Mior Priority: Trivial (was: Major) > Allow the long form of call sites

[jira] [Resolved] (SPARK-24361) Polish code block manipulation API

[ https://issues.apache.org/jira/browse/SPARK-24361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24361. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21405

[jira] [Assigned] (SPARK-24361) Polish code block manipulation API

[ https://issues.apache.org/jira/browse/SPARK-24361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24361: --- Assignee: Liang-Chi Hsieh > Polish code block manipulation API >

[jira] [Comment Edited] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533499#comment-16533499 ] Hari Sekhon edited comment on SPARK-24474 at 7/5/18 10:43 AM: -- My main

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533499#comment-16533499 ] Hari Sekhon commented on SPARK-24474: - My main concern with this workaround is pulling half the

[jira] [Commented] (SPARK-24464) Unit tests for MLlib's Instrumentation

[ https://issues.apache.org/jira/browse/SPARK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533461#comment-16533461 ] Liang-Chi Hsieh commented on SPARK-24464: - Because {{Instrumentation}} is used for logging, I

[jira] [Assigned] (SPARK-24744) Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user.

[ https://issues.apache.org/jira/browse/SPARK-24744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24744: Assignee: Apache Spark > Structured Streaming set SparkSession configuration with the

[jira] [Assigned] (SPARK-24744) Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user.

[ https://issues.apache.org/jira/browse/SPARK-24744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24744: Assignee: (was: Apache Spark) > Structured Streaming set SparkSession configuration

[jira] [Commented] (SPARK-24744) Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user.

[ https://issues.apache.org/jira/browse/SPARK-24744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533429#comment-16533429 ] Apache Spark commented on SPARK-24744: -- User 'bjkonglu' has created a pull request for this issue:

[jira] [Updated] (SPARK-24744) Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user.

[ https://issues.apache.org/jira/browse/SPARK-24744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bjkonglu updated SPARK-24744: - Description: h3. Background When I use structured streaming to construct my application, there is

[jira] [Created] (SPARK-24745) Map function does not keep rdd name

Igor Pergenitsa created SPARK-24745: --- Summary: Map function does not keep rdd name Key: SPARK-24745 URL: https://issues.apache.org/jira/browse/SPARK-24745 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533403#comment-16533403 ] Apache Spark commented on SPARK-24743: -- User 'cluo512' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24743: Assignee: Apache Spark > Update the JavaDirectKafkaWordCount example to support the new

[jira] [Assigned] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24743: Assignee: (was: Apache Spark) > Update the JavaDirectKafkaWordCount example to

[jira] [Comment Edited] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533391#comment-16533391 ] Hyukjin Kwon edited comment on SPARK-24673 at 7/5/18 8:13 AM: -- Fixed in

[jira] [Resolved] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24673. -- Resolution: Fixed Assignee: Antonio Murgia Fix Version/s: 2.4.0 Fixed in

[jira] [Created] (SPARK-24744) Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user.

bjkonglu created SPARK-24744: Summary: Structured Streaming set SparkSession configuration with the value in the metadata if there is not a option set by user. Key: SPARK-24744 URL:

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1659#comment-1659 ] Ohad Raviv commented on SPARK-24528: After digging a little bit in the code and Jira I understand

[jira] [Updated] (SPARK-24743) Update the JavaDirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luochuan updated SPARK-24743: - Summary: Update the JavaDirectKafkaWordCount example to support the new API of Kafka (was: Update the

[jira] [Updated] (SPARK-24743) Update the DirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luochuan updated SPARK-24743: - Description: when I ran the example JavaDirectKafkaWordCount as follows: ./bin/spark-submit --class

[jira] [Created] (SPARK-24743) Update the DirectKafkaWordCount example to support the new API of Kafka

luochuan created SPARK-24743: Summary: Update the DirectKafkaWordCount example to support the new API of Kafka Key: SPARK-24743 URL: https://issues.apache.org/jira/browse/SPARK-24743 Project: Spark

[jira] [Commented] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533286#comment-16533286 ] Kaya Kupferschmidt commented on SPARK-24742: I will provide a patch later. > Field Metadata

[jira] [Created] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

Kaya Kupferschmidt created SPARK-24742: -- Summary: Field Metadata raises NullPointerException in hashCode method Key: SPARK-24742 URL: https://issues.apache.org/jira/browse/SPARK-24742 Project: