[jira] [Commented] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458952#comment-17458952 ] Hyukjin Kwon commented on SPARK-37634: -- Thanks for investigating and confirming that. > Jline

[jira] [Resolved] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37634. -- Resolution: Not A Problem > Jline throw NoSuchMethodError when compiling Spark on windows  >

[jira] [Updated] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37638: - Description: {code} >>> ps.range(10) 21/12/14 16:12:58 WARN SparkSession: Using an existing

[jira] [Commented] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458948#comment-17458948 ] Apache Spark commented on SPARK-37638: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37638: Assignee: Apache Spark (was: Hyukjin Kwon) > Use existing active Spark session instead

[jira] [Commented] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458947#comment-17458947 ] Apache Spark commented on SPARK-37638: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37638: Assignee: Hyukjin Kwon (was: Apache Spark) > Use existing active Spark session instead

[jira] [Updated] (SPARK-37638) Make static and core configuration log less noisy when creating a Spark session

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37638: - Component/s: PySpark (was: SQL) > Make static and core configuration log

[jira] [Updated] (SPARK-37638) Make static and core configuration log less noisy when creating a Spark session

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37638: - Description: {code} >>> ps.range(10) 21/12/14 16:12:58 WARN SparkSession: Using an existing

[jira] [Updated] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37638: - Summary: Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on

[jira] [Updated] (SPARK-37639) spark history server clean event log directory with out check status file

2021-12-13 Thread muhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37639: --- Description: i foud a problem, the thrift server create event log file(.inprogress file create at init),

[jira] [Updated] (SPARK-37641) Support ANSI Aggregate Function: regr_r2

2021-12-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37641: --- Description: REGR_R2 is an ANSI aggregate function. many database support it. > Support ANSI

[jira] [Created] (SPARK-37641) Support ANSI Aggregate Function: regr_r2

2021-12-13 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-37641: -- Summary: Support ANSI Aggregate Function: regr_r2 Key: SPARK-37641 URL: https://issues.apache.org/jira/browse/SPARK-37641 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread JianSheng Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458938#comment-17458938 ] JianSheng Zhang commented on SPARK-37634: - After doing some test, I think it's more about

[jira] [Created] (SPARK-37640) rolled event log still need be clean after compact

2021-12-13 Thread muhong (Jira)
muhong created SPARK-37640: -- Summary: rolled event log still need be clean after compact Key: SPARK-37640 URL: https://issues.apache.org/jira/browse/SPARK-37640 Project: Spark Issue Type:

[jira] [Created] (SPARK-37639) spark history server clean event log directory with out check status file

2021-12-13 Thread muhong (Jira)
muhong created SPARK-37639: -- Summary: spark history server clean event log directory with out check status file Key: SPARK-37639 URL: https://issues.apache.org/jira/browse/SPARK-37639 Project: Spark

[jira] [Commented] (SPARK-28869) Roll over event log files

2021-12-13 Thread muhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458931#comment-17458931 ] muhong commented on SPARK-28869: i foud a problem, the thrift server create event log file(.inprogress

[jira] [Updated] (SPARK-37638) Make static and core configuration log less noisy when creating a Spark session

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37638: - Description: It is too noisy to show warnings every time when a Spark session is created. For

[jira] [Created] (SPARK-37638) Make static and core configuration log less noisy when creating a Spark session

2021-12-13 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37638: Summary: Make static and core configuration log less noisy when creating a Spark session Key: SPARK-37638 URL: https://issues.apache.org/jira/browse/SPARK-37638

[jira] [Commented] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458919#comment-17458919 ] Apache Spark commented on SPARK-37620: -- User 'ByronHsu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37620: Assignee: Apache Spark > Use more precise types for SparkContext Optional fields (i.e.

[jira] [Assigned] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37620: Assignee: (was: Apache Spark) > Use more precise types for SparkContext Optional

[jira] [Commented] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458917#comment-17458917 ] Apache Spark commented on SPARK-37620: -- User 'ByronHsu' has created a pull request for this issue:

[jira] [Created] (SPARK-37637) Migrate SHOW TABLES EXTENDED to use v2 command by default

2021-12-13 Thread Terry Kim (Jira)
Terry Kim created SPARK-37637: - Summary: Migrate SHOW TABLES EXTENDED to use v2 command by default Key: SPARK-37637 URL: https://issues.apache.org/jira/browse/SPARK-37637 Project: Spark Issue

[jira] [Commented] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default

2021-12-13 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458907#comment-17458907 ] Terry Kim commented on SPARK-37636: --- working on this > Migrate CREATE NAMESPACE to use v2 command by

[jira] [Commented] (SPARK-37637) Migrate SHOW TABLES EXTENDED to use v2 command by default

2021-12-13 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458908#comment-17458908 ] Terry Kim commented on SPARK-37637: --- Working on this. > Migrate SHOW TABLES EXTENDED to use v2

[jira] [Commented] (SPARK-37310) Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by default

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458906#comment-17458906 ] Apache Spark commented on SPARK-37310: -- User 'imback82' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37310) Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by default

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37310: Assignee: Apache Spark > Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by

[jira] [Assigned] (SPARK-37310) Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by default

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37310: Assignee: (was: Apache Spark) > Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2

[jira] [Assigned] (SPARK-37310) Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by default

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37310: Assignee: Apache Spark > Migrate ALTER NAMESPACE ... SET PROPERTIES to use v2 command by

[jira] [Created] (SPARK-37636) Migrate CREATE NAMESPACE to use v2 command by default

2021-12-13 Thread Terry Kim (Jira)
Terry Kim created SPARK-37636: - Summary: Migrate CREATE NAMESPACE to use v2 command by default Key: SPARK-37636 URL: https://issues.apache.org/jira/browse/SPARK-37636 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37635) SHOW TBLPROPERTIES should print the fully qualified table name

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458897#comment-17458897 ] Apache Spark commented on SPARK-37635: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-37635) SHOW TBLPROPERTIES should print the fully qualified table name

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458896#comment-17458896 ] Apache Spark commented on SPARK-37635: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37635) SHOW TBLPROPERTIES should print the fully qualified table name

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37635: Assignee: (was: Apache Spark) > SHOW TBLPROPERTIES should print the fully qualified

[jira] [Assigned] (SPARK-37635) SHOW TBLPROPERTIES should print the fully qualified table name

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37635: Assignee: Apache Spark > SHOW TBLPROPERTIES should print the fully qualified table name

[jira] [Created] (SPARK-37635) SHOW TBLPROPERTIES should print the fully qualified table name

2021-12-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-37635: --- Summary: SHOW TBLPROPERTIES should print the fully qualified table name Key: SPARK-37635 URL: https://issues.apache.org/jira/browse/SPARK-37635 Project: Spark

[jira] [Updated] (SPARK-37577) ClassCastException: ArrayType cannot be cast to StructType

2021-12-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37577: -- Fix Version/s: 3.2.1 > ClassCastException: ArrayType cannot be cast to StructType >

[jira] [Commented] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458892#comment-17458892 ] Hyukjin Kwon commented on SPARK-37634: -- cc [~dongjoon] FYI > Jline throw NoSuchMethodError when

[jira] [Commented] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458890#comment-17458890 ] Hyukjin Kwon commented on SPARK-37634: -- Building itself on Windows isn't officially support in

[jira] [Commented] (SPARK-37217) The number of dynamic partitions should early check when writing to external tables

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458891#comment-17458891 ] Apache Spark commented on SPARK-37217: -- User 'cxzl25' has created a pull request for this issue:

[jira] [Updated] (SPARK-37481) Disappearance of skipped stages mislead the bug hunting

2021-12-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-37481: - Fix Version/s: 3.2.1 (was: 3.2.0) > Disappearance of skipped stages mislead the

[jira] [Updated] (SPARK-37575) null values should be saved as nothing rather than quoted empty Strings "" with default settings

2021-12-13 Thread Guo Wei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guo Wei updated SPARK-37575: Summary: null values should be saved as nothing rather than quoted empty Strings "" with default settings

[jira] [Assigned] (SPARK-37217) The number of dynamic partitions should early check when writing to external tables

2021-12-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-37217: Assignee: dzcxzl > The number of dynamic partitions should early check when writing to external

[jira] [Resolved] (SPARK-37217) The number of dynamic partitions should early check when writing to external tables

2021-12-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-37217. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34493

[jira] [Resolved] (SPARK-37631) Code clean up on promoting strings in math functions

2021-12-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-37631. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34884

[jira] [Comment Edited] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2021-12-13 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458866#comment-17458866 ] Mr.黄 edited comment on SPARK-4105 at 12/14/21, 2:39 AM: [~joshrosen] Hello, I

[jira] [Comment Edited] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2021-12-13 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458866#comment-17458866 ] Mr.黄 edited comment on SPARK-4105 at 12/14/21, 2:38 AM: [~joshrosen] Hello, I

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2021-12-13 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458866#comment-17458866 ] Mr.黄 commented on SPARK-4105: - [~joshrosen] Hello, I still encountered this problem in the company's project

[jira] [Created] (SPARK-37634) Jline throw NoSuchMethodError when compiling Spark on windows 

2021-12-13 Thread JianSheng Zhang (Jira)
JianSheng Zhang created SPARK-37634: --- Summary: Jline throw NoSuchMethodError when compiling Spark on windows  Key: SPARK-37634 URL: https://issues.apache.org/jira/browse/SPARK-37634 Project: Spark

[jira] [Assigned] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37633: Assignee: (was: Apache Spark) > Unwrap cast should skip if downcast failed with ansi

[jira] [Assigned] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37633: Assignee: Apache Spark > Unwrap cast should skip if downcast failed with ansi enabled >

[jira] [Commented] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458862#comment-17458862 ] Apache Spark commented on SPARK-37633: -- User 'manuzhang' has created a pull request for this issue:

[jira] [Updated] (SPARK-37518) inject an early scan pushdown rule

2021-12-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37518: --- Summary: inject an early scan pushdown rule (was: inject a early scan pushdown rule) > inject an

[jira] [Created] (SPARK-37633) Unwrap cast should skip if downcast failed with ansi enabled

2021-12-13 Thread Manu Zhang (Jira)
Manu Zhang created SPARK-37633: -- Summary: Unwrap cast should skip if downcast failed with ansi enabled Key: SPARK-37633 URL: https://issues.apache.org/jira/browse/SPARK-37633 Project: Spark

[jira] [Commented] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458826#comment-17458826 ] Hyukjin Kwon commented on SPARK-37621: -- Thanks [~rdblue] for clarification. I have been just

[jira] [Updated] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37621: - Component/s: SQL (was: Input/Output) > ClassCastException when trying to

[jira] [Commented] (SPARK-37625) update log4j to 2.15

2021-12-13 Thread PJ Fanning (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458802#comment-17458802 ] PJ Fanning commented on SPARK-37625: log4j 2.16.0 is out - it might be best to pause this as it

[jira] [Commented] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread PJ Fanning (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458798#comment-17458798 ] PJ Fanning commented on SPARK-37630: Maybe a duplicate of SPARK-6305 > Security issue from Log4j

[jira] [Commented] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458792#comment-17458792 ] Apache Spark commented on SPARK-37632: -- User 'zero323' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37632: Assignee: Apache Spark > Drop code targetting Python < 3.7 >

[jira] [Assigned] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37632: Assignee: (was: Apache Spark) > Drop code targetting Python < 3.7 >

[jira] [Commented] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458791#comment-17458791 ] Apache Spark commented on SPARK-37632: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-37577) ClassCastException: ArrayType cannot be cast to StructType

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458789#comment-17458789 ] Apache Spark commented on SPARK-37577: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-37577) ClassCastException: ArrayType cannot be cast to StructType

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458787#comment-17458787 ] Apache Spark commented on SPARK-37577: -- User 'viirya' has created a pull request for this issue:

[jira] [Updated] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37632: --- Description: We should drop parts of code that target Python 3.6 (or less if these

[jira] [Created] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37632: -- Summary: Drop code targetting Python < 3.7 Key: SPARK-37632 URL: https://issues.apache.org/jira/browse/SPARK-37632 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37609) Transient StackOverflowError on DataFrame from Catalyst QueryPlan

2021-12-13 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458691#comment-17458691 ] Rafal Wojdyla commented on SPARK-37609: --- [~hyukjin.kwon] yep, understand that, if I have some time

[jira] [Resolved] (SPARK-34511) Current Security vulnerabilities in spark libraries

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34511. -- Resolution: Not A Problem > Current Security vulnerabilities in spark libraries >

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37630: - Priority: Major (was: Blocker) (Unsetting Blocker - reserved for committers and would be

[jira] [Commented] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458652#comment-17458652 ] Sean R. Owen commented on SPARK-37630: -- First: updating to log4j 2 would be nice, but is already

[jira] [Resolved] (SPARK-36969) Inline type hints for SparkContext

2021-12-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-36969. Fix Version/s: 3.3.0 Resolution: Duplicate > Inline type hints for

[jira] [Updated] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-37621: Description: I am gettin an error when I try to persist the results on a Join operation. Note

[jira] [Commented] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Ciprian Gerea (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458562#comment-17458562 ] Ciprian Gerea commented on SPARK-37621: --- I was able to force Spark to add the extra projection

[jira] (SPARK-37625) update log4j to 2.15

2021-12-13 Thread Gustavo Martin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37625 ] Gustavo Martin deleted comment on SPARK-37625: was (Author: gumartinm): What about other Spark versions? Will there be any patch? > update log4j to 2.15 > - > >

[jira] [Assigned] (SPARK-36571) Optimized FileOutputCommitter with StagingDir

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36571: Assignee: (was: Apache Spark) > Optimized FileOutputCommitter with StagingDir >

[jira] [Assigned] (SPARK-36571) Optimized FileOutputCommitter with StagingDir

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36571: Assignee: Apache Spark > Optimized FileOutputCommitter with StagingDir >

[jira] [Commented] (SPARK-37621) ClassCastException when trying to persist the result of a join between two Iceberg tables

2021-12-13 Thread Ryan Blue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458516#comment-17458516 ] Ryan Blue commented on SPARK-37621: --- [~hyukjin.kwon], this affects any source that doesn't always

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Description: log4j is being used in version [1.2.17|#L122]]   This version has been

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Description: log4j is being used in version [1.2.17|#L122]]   This version has been

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Description: log4j is being used in version [1.2.17|#L122]]   This version has been

[jira] [Commented] (SPARK-37631) Code clean up on promoting strings in math functions

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458483#comment-17458483 ] Apache Spark commented on SPARK-37631: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-37631) Code clean up on promoting strings in math functions

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37631: Assignee: Apache Spark (was: Gengliang Wang) > Code clean up on promoting strings in

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Description: log4j is being used in version [1.2.17|#L122]]   This version has been

[jira] [Assigned] (SPARK-37631) Code clean up on promoting strings in math functions

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37631: Assignee: Gengliang Wang (was: Apache Spark) > Code clean up on promoting strings in

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Labels: security (was: ) > Security issue from Log4j 0day exploit >

[jira] [Created] (SPARK-37631) Code clean up on promoting strings in math functions

2021-12-13 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-37631: -- Summary: Code clean up on promoting strings in math functions Key: SPARK-37631 URL: https://issues.apache.org/jira/browse/SPARK-37631 Project: Spark

[jira] [Created] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
Ismail Hrimech created SPARK-37630: -- Summary: Security issue from Log4j 0day exploit Key: SPARK-37630 URL: https://issues.apache.org/jira/browse/SPARK-37630 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-37630) Security issue from Log4j 0day exploit

2021-12-13 Thread Ismail Hrimech (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismail Hrimech updated SPARK-37630: --- Priority: Blocker (was: Major) > Security issue from Log4j 0day exploit >

[jira] [Commented] (SPARK-37629) speed up Expression.canonicalized

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458455#comment-17458455 ] Apache Spark commented on SPARK-37629: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37627) Add sorted column in BucketTransform

2021-12-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37627. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34879

[jira] [Assigned] (SPARK-37627) Add sorted column in BucketTransform

2021-12-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37627: --- Assignee: Huaxin Gao > Add sorted column in BucketTransform >

[jira] [Assigned] (SPARK-37629) speed up Expression.canonicalized

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37629: Assignee: (was: Apache Spark) > speed up Expression.canonicalized >

[jira] [Assigned] (SPARK-37629) speed up Expression.canonicalized

2021-12-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37629: Assignee: Apache Spark > speed up Expression.canonicalized >

[jira] [Updated] (SPARK-37493) expose driver gc time and duration time

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37493: - Priority: Minor (was: Major) > expose driver gc time and duration time >

[jira] [Assigned] (SPARK-37493) expose driver gc time and duration time

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37493: Assignee: zhoubin > expose driver gc time and duration time >

[jira] [Resolved] (SPARK-37493) expose driver gc time and duration time

2021-12-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37493. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34749

[jira] [Comment Edited] (SPARK-37625) update log4j to 2.15

2021-12-13 Thread Gustavo Martin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458398#comment-17458398 ] Gustavo Martin edited comment on SPARK-37625 at 12/13/21, 1:49 PM: ---

[jira] [Commented] (SPARK-37625) update log4j to 2.15

2021-12-13 Thread Gustavo Martin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458398#comment-17458398 ] Gustavo Martin commented on SPARK-37625: What about other Spark versions? > update log4j to

[jira] [Created] (SPARK-37629) speed up Expression.canonicalized

2021-12-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-37629: --- Summary: speed up Expression.canonicalized Key: SPARK-37629 URL: https://issues.apache.org/jira/browse/SPARK-37629 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-37591) Support the GCM mode by aes_encrypt()/aes_decrypt()

2021-12-13 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37591. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34852

  1   2   >