[jira] [Created] (SPARK-29610) Keys with Null values are discarded when using to_json function

Jonathan created SPARK-29610: Summary: Keys with Null values are discarded when using to_json function Key: SPARK-29610 URL: https://issues.apache.org/jira/browse/SPARK-29610 Project: Spark

[jira] [Resolved] (SPARK-29033) Always use CreateNamedStructUnsafe, the UnsafeRow-based version of the CreateNamedStruct codepath

[ https://issues.apache.org/jira/browse/SPARK-29033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-29033. Resolution: Won't Fix I'm resolving this as "Won't Fix" because we ended up doing the opposite in

[jira] [Comment Edited] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960244#comment-16960244 ] feiwang edited comment on SPARK-27736 at 10/26/19 2:30 AM: --- Hi, we met this

[jira] [Commented] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960244#comment-16960244 ] feiwang commented on SPARK-27736: - Hi, we met this issue recently. [~joshrosen] [~tgraves] How about

[jira] [Updated] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29608: -- Issue Type: Improvement (was: Task) > Add Hadoop 3.2 profile to binary package >

[jira] [Created] (SPARK-29608) Add Hadoop 3.2 profile to binary package

Dongjoon Hyun created SPARK-29608: - Summary: Add Hadoop 3.2 profile to binary package Key: SPARK-29608 URL: https://issues.apache.org/jira/browse/SPARK-29608 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29608. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26260

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Fix Version/s: (was: 3.0.0) > spark-shell cannot handle `-i` option correctly >

[jira] [Created] (SPARK-29607) Move static methods from CalendarInterval to IntervalUtils

Maxim Gekk created SPARK-29607: -- Summary: Move static methods from CalendarInterval to IntervalUtils Key: SPARK-29607 URL: https://issues.apache.org/jira/browse/SPARK-29607 Project: Spark Issue

[jira] [Assigned] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29608: - Assignee: Dongjoon Hyun > Add Hadoop 3.2 profile to binary package >

[jira] [Created] (SPARK-29609) DataSourceV2: Support DROP NAMESPACE

Terry Kim created SPARK-29609: - Summary: DataSourceV2: Support DROP NAMESPACE Key: SPARK-29609 URL: https://issues.apache.org/jira/browse/SPARK-29609 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-29580) Add kerberos debug messages for Kafka secure tests

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960077#comment-16960077 ] Dongjoon Hyun commented on SPARK-29580: --- We will open a new JIRA if this happens again. For now,

[jira] [Updated] (SPARK-29580) Add kerberos debug messages for Kafka secure tests

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29580: -- Summary: Add kerberos debug messages for Kafka secure tests (was: KafkaDelegationTokenSuite

[jira] [Resolved] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29580. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26252

[jira] [Assigned] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29580: - Assignee: Gabor Somogyi > KafkaDelegationTokenSuite fails to create new

[jira] [Commented] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959962#comment-16959962 ] Dongjoon Hyun commented on SPARK-29580: --- Thank you for taking a look, [~gsomogyi]. +1 for waiting

[jira] [Resolved] (SPARK-29414) HasOutputCol param isSet() property is not preserved after persistence

[ https://issues.apache.org/jira/browse/SPARK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-29414. -- Fix Version/s: 2.4.4 Resolution: Fixed Thanks [~borys.biletskyy], I'll mark this as

[jira] [Resolved] (SPARK-29508) Implicitly cast strings in datetime arithmetic operations

[ https://issues.apache.org/jira/browse/SPARK-29508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29508. -- Resolution: Won't Fix > Implicitly cast strings in datetime arithmetic operations >

[jira] [Commented] (SPARK-29414) HasOutputCol param isSet() property is not preserved after persistence

[ https://issues.apache.org/jira/browse/SPARK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959874#comment-16959874 ] Borys Biletskyy commented on SPARK-29414: - The test passes in v. 2.4.4 and persistence works as

[jira] [Created] (SPARK-29606) Improve EliminateOuterJoin performance

Yuming Wang created SPARK-29606: --- Summary: Improve EliminateOuterJoin performance Key: SPARK-29606 URL: https://issues.apache.org/jira/browse/SPARK-29606 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-28743) YarnShuffleService leads to NodeManager OOM because ChannelOutboundBuffer has too many entries

[ https://issues.apache.org/jira/browse/SPARK-28743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959858#comment-16959858 ] Anton Ippolitov commented on SPARK-28743: - Hi! We are seeing the exact same issue with Spark

[jira] [Updated] (SPARK-28743) YarnShuffleService leads to NodeManager OOM because ChannelOutboundBuffer has too many entries

[ https://issues.apache.org/jira/browse/SPARK-28743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Ippolitov updated SPARK-28743: Attachment: Screen Shot 2019-10-25 at 17.24.10.png > YarnShuffleService leads to

[jira] [Created] (SPARK-29605) Optimize string to interval casting

Maxim Gekk created SPARK-29605: -- Summary: Optimize string to interval casting Key: SPARK-29605 URL: https://issues.apache.org/jira/browse/SPARK-29605 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29574) spark with user provided hadoop doesn't work on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-29574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959849#comment-16959849 ] Michał Wesołowski commented on SPARK-29574: --- I investigated the executor issue. It doesn't

[jira] [Resolved] (SPARK-29527) SHOW CREATE TABLE should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29527. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26184

[jira] [Resolved] (SPARK-29545) Implement bitwise integer aggregates bit_xor

[ https://issues.apache.org/jira/browse/SPARK-29545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29545. -- Resolution: Fixed Resolved by [https://github.com/apache/spark/pull/26205] >

[jira] [Assigned] (SPARK-29500) Support partition column when writing to Kafka

[ https://issues.apache.org/jira/browse/SPARK-29500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29500: Assignee: Nicola Bova > Support partition column when writing to Kafka >

[jira] [Resolved] (SPARK-29500) Support partition column when writing to Kafka

[ https://issues.apache.org/jira/browse/SPARK-29500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29500. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26153

[jira] [Commented] (SPARK-29604) SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set

[ https://issues.apache.org/jira/browse/SPARK-29604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959711#comment-16959711 ] Jungtaek Lim commented on SPARK-29604: -- I've figured out the root cause and have a patch. Will

[jira] [Created] (SPARK-29604) SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set

Jungtaek Lim created SPARK-29604: Summary: SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set Key: SPARK-29604 URL:

[jira] [Created] (SPARK-29603) Support application priority for spark on yarn

Kent Yao created SPARK-29603: Summary: Support application priority for spark on yarn Key: SPARK-29603 URL: https://issues.apache.org/jira/browse/SPARK-29603 Project: Spark Issue Type:

[jira] [Reopened] (SPARK-9612) Add instance weight support for GBTs

[ https://issues.apache.org/jira/browse/SPARK-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reopened SPARK-9612: - Assignee: zhengruifeng (was: DB Tsai) > Add instance weight support for GBTs >

[jira] [Resolved] (SPARK-9612) Add instance weight support for GBTs

[ https://issues.apache.org/jira/browse/SPARK-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-9612. - Resolution: Fixed > Add instance weight support for GBTs >

[jira] [Updated] (SPARK-29599) Support pagination for session table in JDBC/ODBC Tab

[ https://issues.apache.org/jira/browse/SPARK-29599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29599: -- Summary: Support pagination for session table in JDBC/ODBC Tab (was: Support pagination for session

[jira] [Created] (SPARK-29602) How does the spark from_json json and dataframe transform ignore the case of the json key

ruiliang created SPARK-29602: Summary: How does the spark from_json json and dataframe transform ignore the case of the json key Key: SPARK-29602 URL: https://issues.apache.org/jira/browse/SPARK-29602

[jira] [Commented] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959512#comment-16959512 ] Gabor Somogyi commented on SPARK-29580: --- I think

[jira] [Resolved] (SPARK-29461) Spark dataframe writer does not expose metrics for JDBC writer

[ https://issues.apache.org/jira/browse/SPARK-29461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29461. -- Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed

[jira] [Commented] (SPARK-29601) JDBC ODBC Tab Statement column should be provided ellipsis for bigger SQL statement

[ https://issues.apache.org/jira/browse/SPARK-29601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959479#comment-16959479 ] pavithra ramachandran commented on SPARK-29601: --- i shall work on this > JDBC ODBC Tab

[jira] [Created] (SPARK-29601) JDBC ODBC Tab Statement column should be provided ellipsis for bigger SQL statement

ABHISHEK KUMAR GUPTA created SPARK-29601: Summary: JDBC ODBC Tab Statement column should be provided ellipsis for bigger SQL statement Key: SPARK-29601 URL:

[jira] [Commented] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959467#comment-16959467 ] Udbhav Agrawal commented on SPARK-29600: I will check this issue > array_contains built in

[jira] [Created] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

ABHISHEK KUMAR GUPTA created SPARK-29600: Summary: array_contains built in function is not backward compatible in 3.0 Key: SPARK-29600 URL: https://issues.apache.org/jira/browse/SPARK-29600

[jira] [Commented] (SPARK-29599) Support pagination for session table in JDBC/ODBC Session page

[ https://issues.apache.org/jira/browse/SPARK-29599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959454#comment-16959454 ] angerszhu commented on SPARK-29599: --- work on this. > Support pagination for session table in

[jira] [Created] (SPARK-29599) Support pagination for session table in JDBC/ODBC Session page

angerszhu created SPARK-29599: - Summary: Support pagination for session table in JDBC/ODBC Session page Key: SPARK-29599 URL: https://issues.apache.org/jira/browse/SPARK-29599 Project: Spark