[jira] [Assigned] (SPARK-29490) Reset 'WritableColumnVector' in 'RowToColumnarExec'

[ https://issues.apache.org/jira/browse/SPARK-29490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29490: - Assignee: Rong Ma > Reset 'WritableColumnVector' in 'RowToColumnarExec' > -

[jira] [Resolved] (SPARK-29490) Reset 'WritableColumnVector' in 'RowToColumnarExec'

[ https://issues.apache.org/jira/browse/SPARK-29490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29490. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26137 [https://

[jira] [Updated] (SPARK-29554) Add `version` SQL function

[ https://issues.apache.org/jira/browse/SPARK-29554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29554: -- Description: |string|version()|Returns the Hive version (as of Hive 2.1.0). The string contai

[jira] [Assigned] (SPARK-29554) Add a misc function named version

[ https://issues.apache.org/jira/browse/SPARK-29554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29554: - Assignee: Kent Yao > Add a misc function named version > --

[jira] [Resolved] (SPARK-29554) Add a misc function named version

[ https://issues.apache.org/jira/browse/SPARK-29554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29554. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26209 [https://

[jira] [Updated] (SPARK-29554) Add `version` SQL function

[ https://issues.apache.org/jira/browse/SPARK-29554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29554: -- Summary: Add `version` SQL function (was: Add a misc function named version) > Add `version`

[jira] [Created] (SPARK-29610) Keys with Null values are discarded when using to_json function

Jonathan created SPARK-29610: Summary: Keys with Null values are discarded when using to_json function Key: SPARK-29610 URL: https://issues.apache.org/jira/browse/SPARK-29610 Project: Spark Issu

[jira] [Resolved] (SPARK-29033) Always use CreateNamedStructUnsafe, the UnsafeRow-based version of the CreateNamedStruct codepath

[ https://issues.apache.org/jira/browse/SPARK-29033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-29033. Resolution: Won't Fix I'm resolving this as "Won't Fix" because we ended up doing the opposite in

[jira] [Comment Edited] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960244#comment-16960244 ] feiwang edited comment on SPARK-27736 at 10/26/19 2:30 AM: --- Hi

[jira] [Commented] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960244#comment-16960244 ] feiwang commented on SPARK-27736: - Hi, we met this issue recently. [~joshrosen] [~tgrave

[jira] [Updated] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29608: -- Issue Type: Improvement (was: Task) > Add Hadoop 3.2 profile to binary package >

[jira] [Created] (SPARK-29608) Add Hadoop 3.2 profile to binary package

Dongjoon Hyun created SPARK-29608: - Summary: Add Hadoop 3.2 profile to binary package Key: SPARK-29608 URL: https://issues.apache.org/jira/browse/SPARK-29608 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29608. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26260 [https://

[jira] [Updated] (SPARK-25906) spark-shell cannot handle `-i` option correctly

[ https://issues.apache.org/jira/browse/SPARK-25906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25906: -- Fix Version/s: (was: 3.0.0) > spark-shell cannot handle `-i` option correctly > --

[jira] [Created] (SPARK-29607) Move static methods from CalendarInterval to IntervalUtils

Maxim Gekk created SPARK-29607: -- Summary: Move static methods from CalendarInterval to IntervalUtils Key: SPARK-29607 URL: https://issues.apache.org/jira/browse/SPARK-29607 Project: Spark Issue

[jira] [Assigned] (SPARK-29608) Add Hadoop 3.2 profile to binary package

[ https://issues.apache.org/jira/browse/SPARK-29608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29608: - Assignee: Dongjoon Hyun > Add Hadoop 3.2 profile to binary package > --

[jira] [Created] (SPARK-29609) DataSourceV2: Support DROP NAMESPACE

Terry Kim created SPARK-29609: - Summary: DataSourceV2: Support DROP NAMESPACE Key: SPARK-29609 URL: https://issues.apache.org/jira/browse/SPARK-29609 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-29580) Add kerberos debug messages for Kafka secure tests

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960077#comment-16960077 ] Dongjoon Hyun commented on SPARK-29580: --- We will open a new JIRA if this happens a

[jira] [Updated] (SPARK-29580) Add kerberos debug messages for Kafka secure tests

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29580: -- Summary: Add kerberos debug messages for Kafka secure tests (was: KafkaDelegationTokenSuite f

[jira] [Resolved] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29580. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26252 [https://

[jira] [Assigned] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29580: - Assignee: Gabor Somogyi > KafkaDelegationTokenSuite fails to create new KafkaAdminClien

[jira] [Commented] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959962#comment-16959962 ] Dongjoon Hyun commented on SPARK-29580: --- Thank you for taking a look, [~gsomogyi].

[jira] [Resolved] (SPARK-29414) HasOutputCol param isSet() property is not preserved after persistence

[ https://issues.apache.org/jira/browse/SPARK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-29414. -- Fix Version/s: 2.4.4 Resolution: Fixed Thanks [~borys.biletskyy], I'll mark this as res

[jira] [Resolved] (SPARK-29508) Implicitly cast strings in datetime arithmetic operations

[ https://issues.apache.org/jira/browse/SPARK-29508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29508. -- Resolution: Won't Fix > Implicitly cast strings in datetime arithmetic operations > --

[jira] [Commented] (SPARK-29414) HasOutputCol param isSet() property is not preserved after persistence

[ https://issues.apache.org/jira/browse/SPARK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959874#comment-16959874 ] Borys Biletskyy commented on SPARK-29414: - The test passes in v. 2.4.4 and persi

[jira] [Created] (SPARK-29606) Improve EliminateOuterJoin performance

Yuming Wang created SPARK-29606: --- Summary: Improve EliminateOuterJoin performance Key: SPARK-29606 URL: https://issues.apache.org/jira/browse/SPARK-29606 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-28743) YarnShuffleService leads to NodeManager OOM because ChannelOutboundBuffer has too many entries

[ https://issues.apache.org/jira/browse/SPARK-28743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959858#comment-16959858 ] Anton Ippolitov commented on SPARK-28743: - Hi! We are seeing the exact same iss

[jira] [Updated] (SPARK-28743) YarnShuffleService leads to NodeManager OOM because ChannelOutboundBuffer has too many entries

[ https://issues.apache.org/jira/browse/SPARK-28743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Ippolitov updated SPARK-28743: Attachment: Screen Shot 2019-10-25 at 17.24.10.png > YarnShuffleService leads to NodeManag

[jira] [Created] (SPARK-29605) Optimize string to interval casting

Maxim Gekk created SPARK-29605: -- Summary: Optimize string to interval casting Key: SPARK-29605 URL: https://issues.apache.org/jira/browse/SPARK-29605 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29574) spark with user provided hadoop doesn't work on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-29574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959849#comment-16959849 ] Michał Wesołowski commented on SPARK-29574: --- I investigated the executor issue

[jira] [Resolved] (SPARK-29527) SHOW CREATE TABLE should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29527. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26184 [https://gith

[jira] [Resolved] (SPARK-29545) Implement bitwise integer aggregates bit_xor

[ https://issues.apache.org/jira/browse/SPARK-29545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29545. -- Resolution: Fixed Resolved by [https://github.com/apache/spark/pull/26205] > Implemen

[jira] [Assigned] (SPARK-29500) Support partition column when writing to Kafka

[ https://issues.apache.org/jira/browse/SPARK-29500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29500: Assignee: Nicola Bova > Support partition column when writing to Kafka >

[jira] [Resolved] (SPARK-29500) Support partition column when writing to Kafka

[ https://issues.apache.org/jira/browse/SPARK-29500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29500. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26153 [https://gi

[jira] [Commented] (SPARK-29604) SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set

[ https://issues.apache.org/jira/browse/SPARK-29604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959711#comment-16959711 ] Jungtaek Lim commented on SPARK-29604: -- I've figured out the root cause and have a

[jira] [Created] (SPARK-29604) SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set

Jungtaek Lim created SPARK-29604: Summary: SessionState is initialized with isolated classloader for Hive if spark.sql.hive.metastore.jars is being set Key: SPARK-29604 URL: https://issues.apache.org/jira/browse/S

[jira] [Created] (SPARK-29603) Support application priority for spark on yarn

Kent Yao created SPARK-29603: Summary: Support application priority for spark on yarn Key: SPARK-29603 URL: https://issues.apache.org/jira/browse/SPARK-29603 Project: Spark Issue Type: Improvemen

[jira] [Reopened] (SPARK-9612) Add instance weight support for GBTs

[ https://issues.apache.org/jira/browse/SPARK-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reopened SPARK-9612: - Assignee: zhengruifeng (was: DB Tsai) > Add instance weight support for GBTs > ---

[jira] [Resolved] (SPARK-9612) Add instance weight support for GBTs

[ https://issues.apache.org/jira/browse/SPARK-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-9612. - Resolution: Fixed > Add instance weight support for GBTs > >

[jira] [Updated] (SPARK-29599) Support pagination for session table in JDBC/ODBC Tab

[ https://issues.apache.org/jira/browse/SPARK-29599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29599: -- Summary: Support pagination for session table in JDBC/ODBC Tab (was: Support pagination for session

[jira] [Created] (SPARK-29602) How does the spark from_json json and dataframe transform ignore the case of the json key

ruiliang created SPARK-29602: Summary: How does the spark from_json json and dataframe transform ignore the case of the json key Key: SPARK-29602 URL: https://issues.apache.org/jira/browse/SPARK-29602 Pro

[jira] [Commented] (SPARK-29580) KafkaDelegationTokenSuite fails to create new KafkaAdminClient

[ https://issues.apache.org/jira/browse/SPARK-29580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959512#comment-16959512 ] Gabor Somogyi commented on SPARK-29580: --- I think `It_is_not_a_test_it_is_a_sbt_tes

[jira] [Resolved] (SPARK-29461) Spark dataframe writer does not expose metrics for JDBC writer

[ https://issues.apache.org/jira/browse/SPARK-29461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29461. -- Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed Resolved