[jira] [Updated] (SPARK-29629) Support typed integer literal expression

[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29629: - Fix Version/s: (was: 3.0.0) > Support typed integer literal expression >

[jira] [Assigned] (SPARK-29629) Support typed integer literal expression

[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29629: Assignee: (was: Kent Yao) > Support typed integer literal expression >

[jira] [Resolved] (SPARK-29629) Support typed integer literal expression

[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29629. -- Resolution: Won't Fix > Support typed integer literal expression >

[jira] [Reopened] (SPARK-29629) Support typed integer literal expression

[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-29629: -- > Support typed integer literal expression > > >

[jira] [Commented] (SPARK-29629) Support typed integer literal expression

[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999819#comment-16999819 ] Hyukjin Kwon commented on SPARK-29629: -- Reverted at https://github.com/apache/spark/pull/26940 >

[jira] [Updated] (SPARK-30246) Spark on Yarn External Shuffle Service Memory Leak

[ https://issues.apache.org/jira/browse/SPARK-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangweiyi updated SPARK-30246: --- Description: In our large busy yarn cluster which deploy Spark external shuffle service as part of

[jira] [Updated] (SPARK-30246) Spark on Yarn External Shuffle Service Memory Leak

[ https://issues.apache.org/jira/browse/SPARK-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangweiyi updated SPARK-30246: --- Description: In our large busy yarn cluster which deploy Spark external shuffle service as part of

[jira] [Comment Edited] (SPARK-30298) bucket join cannot work for self-join with views

[ https://issues.apache.org/jira/browse/SPARK-30298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999713#comment-16999713 ] Terry Kim edited comment on SPARK-30298 at 12/19/19 6:11 AM: - [~Ankitraj] 

[jira] [Commented] (SPARK-30298) bucket join cannot work for self-join with views

[ https://issues.apache.org/jira/browse/SPARK-30298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999760#comment-16999760 ] Terry Kim commented on SPARK-30298: --- [~Ankitraj] I created a PR: 

[jira] [Updated] (SPARK-30302) Complete info for show create table for views

[ https://issues.apache.org/jira/browse/SPARK-30302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30302: - Description: Add table/column comments and table properties to the result of show create table

[jira] [Updated] (SPARK-30302) Complete info for show create table for views

[ https://issues.apache.org/jira/browse/SPARK-30302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-30302: - Description: Add table/column comments and table properties w > Complete info for show create

[jira] [Created] (SPARK-30302) Complete info for show create table for views

Zhenhua Wang created SPARK-30302: Summary: Complete info for show create table for views Key: SPARK-30302 URL: https://issues.apache.org/jira/browse/SPARK-30302 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30293) In HiveThriftServer2 will remove wrong statement

[ https://issues.apache.org/jira/browse/SPARK-30293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-30293. - Resolution: Not A Bug > In HiveThriftServer2 will remove wrong statement >

[jira] [Commented] (SPARK-30246) Spark on Yarn External Shuffle Service Memory Leak

[ https://issues.apache.org/jira/browse/SPARK-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999730#comment-16999730 ] Jan Filipiak commented on SPARK-30246: -- Hello, we are facing similiiar issues at the moment, hence

[jira] [Comment Edited] (SPARK-30246) Spark on Yarn External Shuffle Service Memory Leak

[ https://issues.apache.org/jira/browse/SPARK-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999730#comment-16999730 ] Jan Filipiak edited comment on SPARK-30246 at 12/19/19 4:26 AM: Hello,

[jira] [Commented] (SPARK-30298) bucket join cannot work for self-join with views

[ https://issues.apache.org/jira/browse/SPARK-30298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999713#comment-16999713 ] Terry Kim commented on SPARK-30298: --- [~Ankitraj] did you already start on this? I faced a similar

[jira] [Commented] (SPARK-29481) all the commands should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999711#comment-16999711 ] Wenchen Fan commented on SPARK-29481: - the v2 API is not able to support everything yet, like view,

[jira] [Created] (SPARK-30301) Datetimes as fields of complex types to hive string results wrong

Kent Yao created SPARK-30301: Summary: Datetimes as fields of complex types to hive string results wrong Key: SPARK-30301 URL: https://issues.apache.org/jira/browse/SPARK-30301 Project: Spark

[jira] [Commented] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999666#comment-16999666 ] Niranjan Artal commented on SPARK-30300: Submitted PR: 

[jira] [Comment Edited] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999641#comment-16999641 ] Niranjan Artal edited comment on SPARK-30300 at 12/19/19 1:56 AM: --

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999650#comment-16999650 ] Hyukjin Kwon commented on SPARK-28502: -- +1 for [~bryanc]'s > Error with struct conversion while

[jira] [Comment Edited] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999641#comment-16999641 ] Niranjan Artal edited comment on SPARK-30300 at 12/19/19 1:15 AM: --

[jira] [Commented] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999641#comment-16999641 ] Niranjan Artal commented on SPARK-30300: [~Ankitraj], Sorry should have updated here.  If you

[jira] [Commented] (SPARK-30298) bucket join cannot work for self-join with views

[ https://issues.apache.org/jira/browse/SPARK-30298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999636#comment-16999636 ] Ankit Raj Boudh commented on SPARK-30298: - I will raise pr for this > bucket join cannot work

[jira] [Commented] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999631#comment-16999631 ] Ankit Raj Boudh commented on SPARK-30300: - [~nartal1], can you please attached snapshot of UI. 

[jira] [Commented] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999629#comment-16999629 ] Ankit Raj Boudh commented on SPARK-30300: - I will start working on this issue  > Update correct

[jira] [Created] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

Niranjan Artal created SPARK-30300: -- Summary: Update correct string in UI for metrics when driver updates same metrics id as tasks. Key: SPARK-30300 URL: https://issues.apache.org/jira/browse/SPARK-30300

[jira] [Resolved] (SPARK-30254) Fix use custom escape lead to LikeSimplification optimize failed

[ https://issues.apache.org/jira/browse/SPARK-30254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30254. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26880

[jira] [Assigned] (SPARK-30254) Fix use custom escape lead to LikeSimplification optimize failed

[ https://issues.apache.org/jira/browse/SPARK-30254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30254: - Assignee: ulysses you > Fix use custom escape lead to LikeSimplification optimize

[jira] [Assigned] (SPARK-30262) Fix NumberFormatException when totalSize is empty

[ https://issues.apache.org/jira/browse/SPARK-30262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30262: - Assignee: chenliang > Fix NumberFormatException when totalSize is empty >

[jira] [Updated] (SPARK-30262) Avoid NumberFormatException when totalSize is empty

[ https://issues.apache.org/jira/browse/SPARK-30262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30262: -- Summary: Avoid NumberFormatException when totalSize is empty (was: Fix

[jira] [Resolved] (SPARK-30262) Fix NumberFormatException when totalSize is empty

[ https://issues.apache.org/jira/browse/SPARK-30262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30262. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26892

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999545#comment-16999545 ] Bryan Cutler commented on SPARK-28502: -- The problem is that returning nested StructTypes is not

[jira] [Resolved] (SPARK-28502) Error with struct conversion while using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-28502. -- Resolution: Fixed > Error with struct conversion while using pandas_udf >

[jira] [Comment Edited] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999454#comment-16999454 ] Abhishek Somani edited comment on SPARK-16996 at 12/18/19 7:43 PM: ---  

[jira] [Comment Edited] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999454#comment-16999454 ] Abhishek Somani edited comment on SPARK-16996 at 12/18/19 7:42 PM: ---  

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999454#comment-16999454 ] Abhishek Somani commented on SPARK-16996: -  [~SandhyaMora] We are [extending the work we did

[jira] [Commented] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999349#comment-16999349 ] Rakesh Raushan commented on SPARK-30288: I am working on it. I will raise the PR soon. > Failed

[jira] [Resolved] (SPARK-29493) Add MapType support for Arrow Java

[ https://issues.apache.org/jira/browse/SPARK-29493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29493. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26512

[jira] [Assigned] (SPARK-29493) Add MapType support for Arrow Java

[ https://issues.apache.org/jira/browse/SPARK-29493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29493: Assignee: Jalpan Randeri > Add MapType support for Arrow Java >

[jira] [Comment Edited] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999237#comment-16999237 ] haiyangyu edited comment on SPARK-30297 at 12/18/19 2:53 PM: -

[jira] [Commented] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999237#comment-16999237 ] haiyangyu commented on SPARK-30297: --- [~r...@databricks.com] [~AMateenM] [~dongjoon] please look this

[jira] [Commented] (SPARK-28315) Decimal can not accept NaN as input

[ https://issues.apache.org/jira/browse/SPARK-28315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999236#comment-16999236 ] Oleg Bonar commented on SPARK-28315: I want to investigate this. > Decimal can not accept NaN as

[jira] [Commented] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999235#comment-16999235 ] haiyangyu commented on SPARK-30297: --- pr here [https://github.com/apache/spark/pull/26938] > Executor

[jira] [Commented] (SPARK-28504) Web UI does not always honor spark.ui.proxyBase and X-Forwarded-Context

[ https://issues.apache.org/jira/browse/SPARK-28504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999214#comment-16999214 ] Jean-Christophe Dauchy commented on SPARK-28504: Tested your implementation, it does not

[jira] [Created] (SPARK-30299) Dynamic allocation with Standalone mode calculates to many executors needed

Thomas Graves created SPARK-30299: - Summary: Dynamic allocation with Standalone mode calculates to many executors needed Key: SPARK-30299 URL: https://issues.apache.org/jira/browse/SPARK-30299

[jira] [Updated] (SPARK-29018) Build spark thrift server on it's own code based on protocol v11

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: h2. Background With the development of Spark and Hive，in current

[jira] [Updated] (SPARK-29018) Build spark thrift server on it's own code based on protocol v11

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: SPIP:Build Spark thrift server based on thrift protocol v11 h2. Background With the

[jira] [Commented] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

[ https://issues.apache.org/jira/browse/SPARK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999194#comment-16999194 ] Ankit Raj Boudh commented on SPARK-30229: - ok, i wii try same operation. >

[jira] [Updated] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haiyangyu updated SPARK-30297: -- Description: h3. *Backgroud* The driver can't sense this executor was lost through the network

[jira] [Created] (SPARK-30298) bucket join cannot work for self-join with views

Xiaoju Wu created SPARK-30298: - Summary: bucket join cannot work for self-join with views Key: SPARK-30298 URL: https://issues.apache.org/jira/browse/SPARK-30298 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haiyangyu updated SPARK-30297: -- Description: h3. *Backgroud* The driver can't sense this executor was lost through the network

[jira] [Updated] (SPARK-30297) Executor heartbeat expired cause app hung up forever

[ https://issues.apache.org/jira/browse/SPARK-30297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haiyangyu updated SPARK-30297: -- Description: h3. *Backgroud* The driver can't sense this executor was lost through the network

[jira] [Created] (SPARK-30297) Executor heartbeat expired cause app hung up forever

haiyangyu created SPARK-30297: - Summary: Executor heartbeat expired cause app hung up forever Key: SPARK-30297 URL: https://issues.apache.org/jira/browse/SPARK-30297 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javier Fuentes updated SPARK-30049: --- Attachment: Screen Shot 2019-12-18 at 9.26.29 AM.png > SQL fails to parse when comment

[jira] [Comment Edited] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999116#comment-16999116 ] Javier Fuentes edited comment on SPARK-30049 at 12/18/19 12:41 PM: ---

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999116#comment-16999116 ] Javier Fuentes commented on SPARK-30049: That issue seems to be fixed in SPARK-30295 [~yumwang] 

[jira] [Updated] (SPARK-29018) Build spark thrift server on it's own code based on protocol v11

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: With the development of Spark and Hive，in current sql/hive-thriftserver module, we need

[jira] [Created] (SPARK-30296) Dataset diffing transformation

Enrico Minack created SPARK-30296: - Summary: Dataset diffing transformation Key: SPARK-30296 URL: https://issues.apache.org/jira/browse/SPARK-30296 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-30295) Remove Hive dependencies from SparkSQLCLI

Javier Fuentes created SPARK-30295: -- Summary: Remove Hive dependencies from SparkSQLCLI Key: SPARK-30295 URL: https://issues.apache.org/jira/browse/SPARK-30295 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29018) Build spark thrift server on it's own code based on protocol v11

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Summary: Build spark thrift server on it's own code based on protocol v11 (was: Spark ThriftServer

[jira] [Issue Comment Deleted] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30288: Comment: was deleted (was: i will start working on this issue) > Failed to write valid

[jira] [Commented] (SPARK-27426) SparkAppHandle states not getting updated in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-27426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999035#comment-16999035 ] jugosag commented on SPARK-27426: - We are observing the exact same problem on Windows with Spark 2.4.4

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998974#comment-16998974 ] Oleg Bonar commented on SPARK-30049: Hi [~yumwang]! Investigating SparkSQLCLIDriver and

[jira] [Commented] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

[ https://issues.apache.org/jira/browse/SPARK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998958#comment-16998958 ] SeaAndHill commented on SPARK-30229: i am using spark streaming to cusum json data from kafka ,and

[jira] [Commented] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998935#comment-16998935 ] Jungtaek Lim commented on SPARK-30294: -- Working on the fix. I might bring the solution first which

[jira] [Created] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

Jungtaek Lim created SPARK-30294: Summary: Read-only state store unnecessarily creates and deletes the temp file for delta file every batch Key: SPARK-30294 URL: https://issues.apache.org/jira/browse/SPARK-30294

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998909#comment-16998909 ] Yuming Wang commented on SPARK-30049: - [~oleg_bonar] Any progress? > SQL fails to parse when