[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304457#comment-17304457 ] Dongjoon Hyun commented on SPARK-27812: --- Thank you for reporting, [~Kotlov]. 1. Co

[jira] [Comment Edited] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304454#comment-17304454 ] Dongjoon Hyun edited comment on SPARK-34785 at 3/18/21, 9:14 PM: -

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304454#comment-17304454 ] Dongjoon Hyun commented on SPARK-34785: --- FYI, each data sources have different sch

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304453#comment-17304453 ] Dongjoon Hyun commented on SPARK-34785: --- SPARK-34212 is complete by itself because

[jira] [Commented] (SPARK-27790) Support ANSI SQL INTERVAL types

2021-03-18 Thread Matthew Powers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304443#comment-17304443 ] Matthew Powers commented on SPARK-27790: [~maxgekk] - how will DateTime addition

[jira] [Commented] (SPARK-34636) sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304434#comment-17304434 ] Apache Spark commented on SPARK-34636: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-34636) sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304433#comment-17304433 ] Apache Spark commented on SPARK-34636: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304427#comment-17304427 ] Apache Spark commented on SPARK-34793: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34793: Assignee: Apache Spark (was: Max Gekk) > Prohibit saving of day-time and year-month inte

[jira] [Commented] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304425#comment-17304425 ] Apache Spark commented on SPARK-34793: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34793: Assignee: Max Gekk (was: Apache Spark) > Prohibit saving of day-time and year-month inte

[jira] [Created] (SPARK-34793) Prohibit saving of day-time and year-month intervals

2021-03-18 Thread Max Gekk (Jira)
Max Gekk created SPARK-34793: Summary: Prohibit saving of day-time and year-month intervals Key: SPARK-34793 URL: https://issues.apache.org/jira/browse/SPARK-34793 Project: Spark Issue Type: Sub-

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304406#comment-17304406 ] Dongjoon Hyun commented on SPARK-24432: --- BTW, ping [~liyinan926] since he is the r

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304404#comment-17304404 ] Dongjoon Hyun commented on SPARK-24432: --- No, [~eje]. This is not closed due to the

[jira] [Updated] (SPARK-34792) Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3

2021-03-18 Thread kondziolka9ld (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kondziolka9ld updated SPARK-34792: -- Description: Hi,  Please consider a following difference of `randomSplit` method even despite

[jira] [Created] (SPARK-34792) Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3

2021-03-18 Thread kondziolka9ld (Jira)
kondziolka9ld created SPARK-34792: - Summary: Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3 Key: SPARK-34792 URL: https://issues.apache.org/jira/browse/SPARK-34792 Project: Spar

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304397#comment-17304397 ] Erik Erlandson commented on SPARK-24432: [~dongjoon] should this be closed, now

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304316#comment-17304316 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304315#comment-17304315 ] Apache Spark commented on SPARK-33482: -- User 'peter-toth' has created a pull reques

[jira] [Updated] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-18 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Chen updated SPARK-34780: - Description: When a dataframe is cached, the logical plan can contain copies of the spark sessi

[jira] [Updated] (SPARK-34780) Cached Table (parquet) with old Configs Used

2021-03-18 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Chen updated SPARK-34780: - Affects Version/s: 3.1.1 > Cached Table (parquet) with old Configs Used > --

[jira] [Updated] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeet updated SPARK-34791: - Description: SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3

[jira] [Updated] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeet updated SPARK-34791: - Description: SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3

[jira] [Created] (SPARK-34791) SparkR throws node stack overflow

2021-03-18 Thread Jeet (Jira)
Jeet created SPARK-34791: Summary: SparkR throws node stack overflow Key: SPARK-34791 URL: https://issues.apache.org/jira/browse/SPARK-34791 Project: Spark Issue Type: Question Components:

[jira] [Commented] (SPARK-34693) Support null in conversions to and from Arrow

2021-03-18 Thread Laurens (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304266#comment-17304266 ] Laurens commented on SPARK-34693: - {code:java} def tps(df) -> pd.DataFrame["workflow_id"

[jira] [Updated] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34789: --- Description: This came up during https://github.com/apache/spark/pull/31877#discuss

[jira] [Created] (SPARK-34790) Fail in fetch shuffle blocks in batch when i/o encryption is enabled.

2021-03-18 Thread hezuojiao (Jira)
hezuojiao created SPARK-34790: - Summary: Fail in fetch shuffle blocks in batch when i/o encryption is enabled. Key: SPARK-34790 URL: https://issues.apache.org/jira/browse/SPARK-34790 Project: Spark

[jira] [Commented] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304245#comment-17304245 ] Attila Zsolt Piros commented on SPARK-34789: I am working on this > Introd

[jira] [Created] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

2021-03-18 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-34789: -- Summary: Introduce Jetty based construct for integration tests where HTTP(S) is used Key: SPARK-34789 URL: https://issues.apache.org/jira/browse/SPARK-34789

[jira] [Resolved] (SPARK-34760) run JavaSQLDataSourceExample failed with Exception in runBasicDataSourceExample().

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-34760. -- Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-34760) run JavaSQLDataSourceExample failed with Exception in runBasicDataSourceExample().

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-34760: Assignee: zengrui > run JavaSQLDataSourceExample failed with Exception in > runBasicDataSourceEx

[jira] [Updated] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-34788: - Component/s: Shuffle > Spark throws FileNotFoundException instead of IOException when disk is full > ---

[jira] [Commented] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304185#comment-17304185 ] wuyi commented on SPARK-34788: -- [~wuxiaofei] Could you take a look at this? > Spark throws

[jira] [Created] (SPARK-34788) Spark throws FileNotFoundException instead of IOException when disk is full

2021-03-18 Thread wuyi (Jira)
wuyi created SPARK-34788: Summary: Spark throws FileNotFoundException instead of IOException when disk is full Key: SPARK-34788 URL: https://issues.apache.org/jira/browse/SPARK-34788 Project: Spark

[jira] [Assigned] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34787: Assignee: (was: Apache Spark) > Option variable in Spark historyServer log should be

[jira] [Assigned] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34787: Assignee: Apache Spark > Option variable in Spark historyServer log should be displayed a

[jira] [Commented] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304145#comment-17304145 ] Apache Spark commented on SPARK-34787: -- User 'kyoty' has created a pull request for

[jira] [Created] (SPARK-34787) Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

2021-03-18 Thread akiyamaneko (Jira)
akiyamaneko created SPARK-34787: --- Summary: Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX) Key: SPARK-34787 URL: https://issues.apache.org/jira/browse/SPARK-34787

[jira] [Commented] (SPARK-34786) read parquet uint64 as decimal

2021-03-18 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304136#comment-17304136 ] Kent Yao commented on SPARK-34786: -- I'd like to do this. > read parquet uint64 as deci

[jira] [Updated] (SPARK-34766) Do not capture maven config for views

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34766: Fix Version/s: 3.1.2 > Do not capture maven config for views > ---

[jira] [Assigned] (SPARK-34774) The `change-scala- version.sh` script not replaced scala.version property correctly

2021-03-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34774: Assignee: Yang Jie > The `change-scala- version.sh` script not replaced scala.version pro

[jira] [Resolved] (SPARK-34774) The `change-scala- version.sh` script not replaced scala.version property correctly

2021-03-18 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34774. -- Fix Version/s: 3.0.3 3.1.2 3.2.0 Resolution: Fixed

[jira] [Created] (SPARK-34786) read parquet uint64 as decimal

2021-03-18 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-34786: --- Summary: read parquet uint64 as decimal Key: SPARK-34786 URL: https://issues.apache.org/jira/browse/SPARK-34786 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Updated] (SPARK-34771) Support UDT for Pandas with Arrow Optimization

2021-03-18 Thread Darcy Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-34771: --- Description: {code:python} spark.conf.set("spark.sql.execution.arrow.enabled", "true") from pyspark.

[jira] [Assigned] (SPARK-34741) MergeIntoTable should avoid ambiguous reference

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34741: --- Assignee: wuyi > MergeIntoTable should avoid ambiguous reference >

[jira] [Resolved] (SPARK-34741) MergeIntoTable should avoid ambiguous reference

2021-03-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34741. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31835 [https://gith

[jira] [Commented] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303904#comment-17303904 ] jobit mathew commented on SPARK-34785: -- [~dongjoon] can you check it once. > SPARK

[jira] [Created] (SPARK-34785) SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file.

2021-03-18 Thread jobit mathew (Jira)
jobit mathew created SPARK-34785: Summary: SPARK-34212 issue not fixed if spark.sql.parquet.enableVectorizedReader=true which is default value. Error Parquet column cannot be converted in file. Key: SPARK-34785 U

[jira] [Resolved] (SPARK-32061) potential regression if use memoryUsage instead of numRows

2021-03-18 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-32061. -- Resolution: Resolved > potential regression if use memoryUsage instead of numRows > --

[jira] [Commented] (SPARK-34087) a memory leak occurs when we clone the spark session

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303895#comment-17303895 ] Apache Spark commented on SPARK-34087: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-34766) Do not capture maven config for views

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303893#comment-17303893 ] Apache Spark commented on SPARK-34766: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-34762) Many PR's Scala 2.13 build action failed

2021-03-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303892#comment-17303892 ] Apache Spark commented on SPARK-34762: -- User 'sarutak' has created a pull request f

<    1   2