[jira] [Commented] (SPARK-34762) Many PR's Scala 2.13 build action failed

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303892#comment-17303892 ] Apache Spark commented on SPARK-34762: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-34766) Do not capture maven config for views

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303893#comment-17303893 ] Apache Spark commented on SPARK-34766: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-34087) a memory leak occurs when we clone the spark session

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303895#comment-17303895 ] Apache Spark commented on SPARK-34087: -- User 'Ngone51' has created a pull request f

[jira] [Resolved] (SPARK-32061) potential regression if use memoryUsage instead of numRows

2021-03-18 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-32061. -- Resolution: Resolved > potential regression if use memoryUsage instead of numRows > --

[jira] [Created] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread jobit mathew (Jira)
jobit mathew created SPARK-34785: Summary: SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file. Key: SPARK-34785 U

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303904#comment-17303904 ] jobit mathew commented on SPARK-34785: -- [~dongjoon] can you check it once. > SPARK

[jira] [Resolved] (SPARK-34741) MergeIntoTable should avoid ambiguous reference

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34741. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31835 [https://gith

[jira] [Assigned] (SPARK-34741) MergeIntoTable should avoid ambiguous reference

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34741: --- Assignee: wuyi > MergeIntoTable should avoid ambiguous reference >

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Created] (SPARK-34786) read parquet uint64 as decimal

2021-03-18 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-34786: --- Summary: read parquet uint64 as decimal Key: SPARK-34786 URL: https://issues.apache.org/jira/browse/SPARK-34786 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-34774) The `change-scala- version.sh` script not replaced scala.version property correctly

2021-03-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34774. -- Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed

[jira] [Assigned] (SPARK-34774) The `change-scala- version.sh` script not replaced scala.version property correctly

2021-03-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34774: Assignee: Yang Jie > The `change-scala- version.sh` script not replaced scala.version pro

[jira] [Updated] (SPARK-34766) Do not capture maven config for views

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34766: Fix Version/s: 3.1.2 > Do not capture maven config for views > ---

[jira] [Commented] (SPARK-34786) read parquet uint64 as decimal

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304136#comment-17304136 ] Kent Yao commented on SPARK-34786: -- I'd like to do this. > read parquet uint64 as deci

[jira] [Created] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread akiyamaneko (Jira)
akiyamaneko created SPARK-34787: --- Summary: Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX) Key: SPARK-34787 URL: https://issues.apache.org/jira/browse/SPARK-34787

[jira] [Commented] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304145#comment-17304145 ] Apache Spark commented on SPARK-34787: -- User 'kyoty' has created a pull request for

[jira] [Assigned] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34787: Assignee: Apache Spark > Option variable in Spark historyServer log should be displayed a

[jira] [Assigned] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34787: Assignee: (was: Apache Spark) > Option variable in Spark historyServer log should be

[jira] [Created] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
wuyi created SPARK-34788: Summary: Spark throws FileNotFoundException instead of IOException when disk is full Key: SPARK-34788 URL: https://issues.apache.org/jira/browse/SPARK-34788 Project: Spark

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304185#comment-17304185 ] wuyi commented on SPARK-34788: -- [~wuxiaofei] Could you take a look at this? > Spark throws

[jira] [Updated] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-34788: - Component/s: Shuffle > Spark throws FileNotFoundException instead of IOException when disk is full > ---

[jira] [Assigned] (SPARK-34760) run JavaSQLDataSourceExample failed with Exception in runBasicDataSourceExample().

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-34760: Assignee: zengrui > run JavaSQLDataSourceExample failed with Exception in > runBasicDataSourceEx

[jira] [Resolved] (SPARK-34760) run JavaSQLDataSourceExample failed with Exception in runBasicDataSourceExample().

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-34760. -- Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed Issue

[jira] [Created] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-34789: -- Summary: Introduce Jetty based construct for integration tests where HTTP(S) is used Key: SPARK-34789 URL: https://issues.apache.org/jira/browse/SPARK-34789

[jira] [Commented] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304245#comment-17304245 ] Attila Zsolt Piros commented on SPARK-34789: I am working on this > Introd

[jira] [Created] (SPARK-34790) Fail in fetch shuffle blocks in batch when i/o encryption is enabled.

2021-03-18 Thread hezuojiao (Jira)
hezuojiao created SPARK-34790: - Summary: Fail in fetch shuffle blocks in batch when i/o encryption is enabled. Key: SPARK-34790 URL: https://issues.apache.org/jira/browse/SPARK-34790 Project: Spark

[jira] [Updated] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34789: --- Description: This came up during https://github.com/apache/spark/pull/31877#discuss

[jira] [Commented] (SPARK-34693) Support null in conversions to and from Arrow

2021-03-18 Thread Laurens (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304266#comment-17304266 ] Laurens commented on SPARK-34693: - {code:java} def tps(df) -> pd.DataFrame["workflow_id"

[jira] [Created] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
Jeet created SPARK-34791: Summary: SparkR throws node stack overflow Key: SPARK-34791 URL: https://issues.apache.org/jira/browse/SPARK-34791 Project: Spark Issue Type: Question Components:

[jira] [Updated] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeet updated SPARK-34791: - Description: SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3

[jira] [Updated] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeet updated SPARK-34791: - Description: SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3

[jira] [Updated] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-18 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Chen updated SPARK-34780: - Affects Version/s: 3.1.1 > Cached Table (parquet) with old Configs Used > --

[jira] [Updated] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-18 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Chen updated SPARK-34780: - Description: When a dataframe is cached, the logical plan can contain copies of the spark sessi

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304315#comment-17304315 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304316#comment-17304316 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304397#comment-17304397 ] Erik Erlandson commented on SPARK-24432: [~dongjoon] should this be closed, now

[jira] [Created] (SPARK-34792) Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3

2021-03-18 Thread kondziolka9ld (Jira)
kondziolka9ld created SPARK-34792: - Summary: Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3 Key: SPARK-34792 URL: https://issues.apache.org/jira/browse/SPARK-34792 Project: Spar

[jira] [Updated] (SPARK-34792) Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3

2021-03-18 Thread kondziolka9ld (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kondziolka9ld updated SPARK-34792: -- Description: Hi,  Please consider a following difference of `randomSplit` method even despite

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304404#comment-17304404 ] Dongjoon Hyun commented on SPARK-24432: --- No, [~eje]. This is not closed due to the

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304406#comment-17304406 ] Dongjoon Hyun commented on SPARK-24432: --- BTW, ping [~liyinan926] since he is the r

[jira] [Created] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Max Gekk (Jira)
Max Gekk created SPARK-34793: Summary: Prohibit saving of day-time and year-month intervals Key: SPARK-34793 URL: https://issues.apache.org/jira/browse/SPARK-34793 Project: Spark Issue Type: Sub-

[jira] [Assigned] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34793: Assignee: Max Gekk (was: Apache Spark) > Prohibit saving of day-time and year-month inte

[jira] [Commented] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304425#comment-17304425 ] Apache Spark commented on SPARK-34793: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34793: Assignee: Apache Spark (was: Max Gekk) > Prohibit saving of day-time and year-month inte

[jira] [Commented] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304427#comment-17304427 ] Apache Spark commented on SPARK-34793: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-34636) sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304433#comment-17304433 ] Apache Spark commented on SPARK-34636: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-34636) sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304434#comment-17304434 ] Apache Spark commented on SPARK-34636: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-27790) Support ANSI SQL INTERVAL types

2021-03-18 Thread Matthew Powers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304443#comment-17304443 ] Matthew Powers commented on SPARK-27790: [~maxgekk] - how will DateTime addition

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304453#comment-17304453 ] Dongjoon Hyun commented on SPARK-34785: --- SPARK-34212 is complete by itself because

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304454#comment-17304454 ] Dongjoon Hyun commented on SPARK-34785: --- FYI, each data sources have different sch

[jira] [Comment Edited] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304454#comment-17304454 ] Dongjoon Hyun edited comment on SPARK-34785 at 3/18/21, 9:14 PM: -

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304457#comment-17304457 ] Dongjoon Hyun commented on SPARK-27812: --- Thank you for reporting, [~Kotlov]. 1. Co

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304459#comment-17304459 ] Dongjoon Hyun commented on SPARK-27812: --- Does your example work with any Spark ver

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304470#comment-17304470 ] Dongjoon Hyun commented on SPARK-27812: --- I found your JIRA, SPARK-34674 . Please c

[jira] [Reopened] (SPARK-34674) Spark app on k8s doesn't terminate without call to sparkContext.stop() method

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-34674: --- I reopen this because the affected version is not the same. The following is a copy of my comme

[jira] [Comment Edited] (SPARK-34674) Spark app on k8s doesn't terminate without call to sparkContext.stop() method

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304472#comment-17304472 ] Dongjoon Hyun edited comment on SPARK-34674 at 3/18/21, 9:39 PM: -

[jira] [Created] (SPARK-34794) Nested higher-order functions broken in DSL

2021-03-18 Thread Daniel Solow (Jira)
Daniel Solow created SPARK-34794: Summary: Nested higher-order functions broken in DSL Key: SPARK-34794 URL: https://issues.apache.org/jira/browse/SPARK-34794 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-34792) Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34792. --- Resolution: Not A Problem Hi, [~kondziolka9ld]. This doesn't look like a bug. It can be dif

[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304504#comment-17304504 ] Dongjoon Hyun commented on SPARK-34791: --- Thank you for reporting, [~jeetendray]. C

[jira] [Commented] (SPARK-34790) Fail in fetch shuffle blocks in batch when i/o encryption is enabled.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304507#comment-17304507 ] Dongjoon Hyun commented on SPARK-34790: --- Thank you for reporting, [~hezuojiao]. Ho

[jira] [Updated] (SPARK-34790) Fail in fetch shuffle blocks in batch when i/o encryption is enabled.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34790: -- Parent: SPARK-33828 Issue Type: Sub-task (was: Bug) > Fail in fetch shuffle blocks in

[jira] [Commented] (SPARK-34790) Fail in fetch shuffle blocks in batch when i/o encryption is enabled.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304508#comment-17304508 ] Dongjoon Hyun commented on SPARK-34790: --- According to the issue title, I convert t

[jira] [Updated] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34789: -- Issue Type: Improvement (was: Task) > Introduce Jetty based construct for integration tests w

[jira] [Commented] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304509#comment-17304509 ] Dongjoon Hyun commented on SPARK-34789: --- Thank you for working on this, [~attilapi

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304511#comment-17304511 ] Dongjoon Hyun commented on SPARK-34788: --- Could you be more specific, [~Ngone51]? I

[jira] [Updated] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34788: -- Affects Version/s: (was: 2.4.7) (was: 3.1.0)

[jira] [Updated] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34787: -- Affects Version/s: (was: 3.0.1) 3.2.0 > Option variable in Spark hi

[jira] [Created] (SPARK-34795) Adds a new job in GitHub Actions to check the output of TPC-DS queries

2021-03-18 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-34795: Summary: Adds a new job in GitHub Actions to check the output of TPC-DS queries Key: SPARK-34795 URL: https://issues.apache.org/jira/browse/SPARK-34795 Projec

[jira] [Updated] (SPARK-34795) Adds a new job in GitHub Actions to check the output of TPC-DS queries

2021-03-18 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34795: - Description: This ticket aims at adding a new job in GitHub Actions to check the output

[jira] [Commented] (SPARK-34795) Adds a new job in GitHub Actions to check the output of TPC-DS queries

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304521#comment-17304521 ] Apache Spark commented on SPARK-34795: -- User 'maropu' has created a pull request fo

[jira] [Assigned] (SPARK-34795) Adds a new job in GitHub Actions to check the output of TPC-DS queries

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34795: Assignee: Apache Spark > Adds a new job in GitHub Actions to check the output of TPC-DS q

[jira] [Assigned] (SPARK-34795) Adds a new job in GitHub Actions to check the output of TPC-DS queries

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34795: Assignee: (was: Apache Spark) > Adds a new job in GitHub Actions to check the output

[jira] [Commented] (SPARK-27790) Support ANSI SQL INTERVAL types

2021-03-18 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304526#comment-17304526 ] Simeon Simeonov commented on SPARK-27790: - Maxim, this is good stuff.  Does ANS

[jira] [Assigned] (SPARK-34794) Nested higher-order functions broken in DSL

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34794: Assignee: Apache Spark > Nested higher-order functions broken in DSL > --

[jira] [Assigned] (SPARK-34794) Nested higher-order functions broken in DSL

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34794: Assignee: (was: Apache Spark) > Nested higher-order functions broken in DSL > ---

[jira] [Commented] (SPARK-34794) Nested higher-order functions broken in DSL

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304532#comment-17304532 ] Apache Spark commented on SPARK-34794: -- User 'dmsolow' has created a pull request f

[jira] [Commented] (SPARK-34794) Nested higher-order functions broken in DSL

2021-03-18 Thread Daniel Solow (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304533#comment-17304533 ] Daniel Solow commented on SPARK-34794: -- PR is at https://github.com/apache/spark/pu

[jira] [Commented] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304538#comment-17304538 ] Apache Spark commented on SPARK-34596: -- User 'maropu' has created a pull request fo

[jira] [Commented] (SPARK-34596) NewInstance.doGenCode should not throw malformed class name error

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304539#comment-17304539 ] Apache Spark commented on SPARK-34596: -- User 'maropu' has created a pull request fo

[jira] [Resolved] (SPARK-34781) Eliminate LEFT SEMI/ANTI join to its left child side with AQE

2021-03-18 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-34781. -- Fix Version/s: 3.2.0 Assignee: Cheng Su Resolution: Fixed Resolved by

[jira] [Created] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Cheng Su (Jira)
Cheng Su created SPARK-34796: Summary: Codegen compilation error for query with LIMIT operator and without AQE Key: SPARK-34796 URL: https://issues.apache.org/jira/browse/SPARK-34796 Project: Spark

[jira] [Commented] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304550#comment-17304550 ] Cheng Su commented on SPARK-34796: -- FYI I am working on a PR to fix it now. > Codegen

[jira] [Updated] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-34796: - Description: Example (reproduced in unit test):   {code:java} test("failed limit query") { withTa

[jira] [Updated] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-34796: - Description: Example (reproduced in unit test):   {code:java} test("failed limit query") { withTa

[jira] [Commented] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304554#comment-17304554 ] Hyukjin Kwon commented on SPARK-34796: -- [~chengsu] I am lowering the priority becau

[jira] [Updated] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34796: - Priority: Critical (was: Blocker) > Codegen compilation error for query with LIMIT operator and

[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304555#comment-17304555 ] Hyukjin Kwon commented on SPARK-34791: -- It's possibly a regression from SPARK-29777

[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304558#comment-17304558 ] Hyukjin Kwon commented on SPARK-34791: -- cc [~falaki] FYI > SparkR throws node stac

[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304556#comment-17304556 ] Hyukjin Kwon commented on SPARK-34791: -- See also https://github.com/apache/spark/pu

[jira] [Commented] (SPARK-34796) Codegen compilation error for query with LIMIT operator and without AQE

2021-03-18 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304559#comment-17304559 ] Cheng Su commented on SPARK-34796: -- [~hyukjin.kwon] - yeah I agree with it. Thanks for

[jira] [Assigned] (SPARK-31661) Document usage of blockSize

2021-03-18 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-31661: Assignee: zhengruifeng > Document usage of blockSize > --- > >

[jira] [Assigned] (SPARK-31976) use MemoryUsage to control the size of block

2021-03-18 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-31976: Assignee: zhengruifeng > use MemoryUsage to control the size of block > -

[jira] [Commented] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304561#comment-17304561 ] Hyukjin Kwon commented on SPARK-34780: -- cc [~sunchao] [~maxgekk] FYI > Cached Tabl

[jira] [Commented] (SPARK-34782) Not equal functions (!=, <>) are missing in SQL doc

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304560#comment-17304560 ] Hyukjin Kwon commented on SPARK-34782: -- Duplicate of SPARK-34747? > Not equal func

[jira] [Resolved] (SPARK-34747) Add virtual operators to the built-in function document.

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34747. -- Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed

[jira] [Commented] (SPARK-34754) sparksql 'add jar' not support hdfs ha mode in k8s

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304568#comment-17304568 ] Hyukjin Kwon commented on SPARK-34754: -- [~lithiumlee-_-] can you test if it works i

[jira] [Commented] (SPARK-34600) Support user defined types in Pandas UDF

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304569#comment-17304569 ] Hyukjin Kwon commented on SPARK-34600: -- [~eddyxu] if this ticket is an umbrella tic

[jira] [Commented] (SPARK-34776) Catalyst error on on certain struct operation (Couldn't find _gen_alias_)

2021-03-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304571#comment-17304571 ] Hyukjin Kwon commented on SPARK-34776: -- cc [~viirya] FYI > Catalyst error on on ce

  1   2   >