[jira] [Created] (SPARK-37285) Add Weight of Evidence and Information value to ml.feature

2021-11-10 Thread Simon Tao (Jira)
Simon Tao created SPARK-37285: - Summary: Add Weight of Evidence and Information value to ml.feature Key: SPARK-37285 URL: https://issues.apache.org/jira/browse/SPARK-37285 Project: Spark Issue Ty

[jira] [Updated] (SPARK-37274) When the value of this parameter is greater than the maximum value of int type, the value will be thrown out of bounds. The document description of this parameter should

2021-11-10 Thread hao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated SPARK-37274: Summary: When the value of this parameter is greater than the maximum value of int type, the value will be thrown

[jira] [Updated] (SPARK-37274) When the value of this parameter is greater than the maximum value of int type, the value will be thrown out of bounds. The document description of this parameter should

2021-11-10 Thread hao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated SPARK-37274: Description: When the value of this parameter is greater than the maximum value of int type, the value will be thr

[jira] [Assigned] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37282: - Assignee: Dongjoon Hyun > Add ExtendedLevelDBTest and disable LevelDB tests on Apple Si

[jira] [Resolved] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37282. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34548 [https://

[jira] [Assigned] (SPARK-37284) Upgrade Jekyll to 4.2.1

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37284: Assignee: Apache Spark (was: Kousuke Saruta) > Upgrade Jekyll to 4.2.1 > ---

[jira] [Assigned] (SPARK-37284) Upgrade Jekyll to 4.2.1

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37284: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade Jekyll to 4.2.1 > ---

[jira] [Commented] (SPARK-37284) Upgrade Jekyll to 4.2.1

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442114#comment-17442114 ] Apache Spark commented on SPARK-37284: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-37284) Upgrade Jekyll to 4.2.1

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442113#comment-17442113 ] Apache Spark commented on SPARK-37284: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-37263) Add PandasAPIOnSparkAdviceWarning class

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37263: Assignee: Apache Spark > Add PandasAPIOnSparkAdviceWarning class > --

[jira] [Created] (SPARK-37284) Upgrade Jekyll to 4.2.1

2021-11-10 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37284: -- Summary: Upgrade Jekyll to 4.2.1 Key: SPARK-37284 URL: https://issues.apache.org/jira/browse/SPARK-37284 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37263) Add PandasAPIOnSparkAdviceWarning class

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442112#comment-17442112 ] Apache Spark commented on SPARK-37263: -- User 'itholic' has created a pull request f

[jira] [Assigned] (SPARK-37263) Add PandasAPIOnSparkAdviceWarning class

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37263: Assignee: (was: Apache Spark) > Add PandasAPIOnSparkAdviceWarning class > ---

[jira] [Commented] (SPARK-37283) Don't try to store a V1 table which contains ANSI intervals in Hive compatible format

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442111#comment-17442111 ] Apache Spark commented on SPARK-37283: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-37283) Don't try to store a V1 table which contains ANSI intervals in Hive compatible format

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37283: Assignee: Apache Spark (was: Kousuke Saruta) > Don't try to store a V1 table which conta

[jira] [Assigned] (SPARK-37283) Don't try to store a V1 table which contains ANSI intervals in Hive compatible format

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37283: Assignee: Kousuke Saruta (was: Apache Spark) > Don't try to store a V1 table which conta

[jira] [Updated] (SPARK-37263) Add PandasAPIOnSparkAdviceWarning class

2021-11-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37263: Description: Raised from comment [https://github.com/apache/spark/pull/34389#discussion_r74173302

[jira] [Updated] (SPARK-37263) Add PandasAPIOnSparkAdviceWarning class

2021-11-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37263: Summary: Add PandasAPIOnSparkAdviceWarning class (was: Add an option to silence advice for pandas

[jira] [Updated] (SPARK-37283) Don't try to store a V1 table which contains ANSI intervals in Hive compatible format

2021-11-10 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37283: --- Description: If, a table being created contains a column of ANSI interval types and the und

[jira] [Created] (SPARK-37283) Don't try to store a V1 table which contains ANSI intervals in Hive compatible format

2021-11-10 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37283: -- Summary: Don't try to store a V1 table which contains ANSI intervals in Hive compatible format Key: SPARK-37283 URL: https://issues.apache.org/jira/browse/SPARK-37283

[jira] [Assigned] (SPARK-37274) These parameters should be of type long, not int

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37274: Assignee: Apache Spark > These parameters should be of type long, not int > -

[jira] [Commented] (SPARK-37274) These parameters should be of type long, not int

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442102#comment-17442102 ] Apache Spark commented on SPARK-37274: -- User 'dh20' has created a pull request for

[jira] [Assigned] (SPARK-37274) These parameters should be of type long, not int

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37274: Assignee: (was: Apache Spark) > These parameters should be of type long, not int > --

[jira] [Updated] (SPARK-37263) Add an option to silence advice for pandas API on Spark.

2021-11-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37263: Description: Raised from comment [https://github.com/apache/spark/pull/34389#discussion_r74173302

[jira] [Updated] (SPARK-37263) Add an option to silence advice for pandas API on Spark.

2021-11-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37263: Summary: Add an option to silence advice for pandas API on Spark. (was: Create an option to silen

[jira] [Updated] (SPARK-37263) Create an option to silence advice for pandas API on Spark.

2021-11-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-37263: Summary: Create an option to silence advice for pandas API on Spark. (was: Reduce pandas-on-Spark

[jira] [Updated] (SPARK-37276) Support YearMonthIntervalType in Arrow

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37276: - Description: Implements the support of YearMonthIntervalType in Arrow code path: - pandas UDFs -

[jira] [Updated] (SPARK-37278) Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37278: - Description: Implements the support of YearMonthIntervalType in: - Python UDFs - createDataFrame

[jira] [Updated] (SPARK-37278) Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37278: - Description: Implements the support of YearMonthIntervalType in: - Python UDFs - createDataFrame

[jira] [Updated] (SPARK-37276) Support YearMonthIntervalType in Arrow

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37276: - Description: Implements the support of YearMonthIntervalType in Arrow code path: - pandas UDFs -

[jira] [Assigned] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37282: Assignee: (was: Apache Spark) > Add ExtendedLevelDBTest and disable LevelDB tests on

[jira] [Assigned] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37282: Assignee: Apache Spark > Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silic

[jira] [Commented] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442084#comment-17442084 ] Apache Spark commented on SPARK-37282: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-37282) Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37282: - Summary: Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon Key: SPARK-37282 URL: https://issues.apache.org/jira/browse/SPARK-37282 Project: Spark

[jira] [Assigned] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-11-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36073: --- Assignee: Peter Toth > EquivalentExpressions fixes and improvements > -

[jira] [Resolved] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-11-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36073. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33281 [https://gith

[jira] [Resolved] (SPARK-36182) Support TimestampNTZ type in Parquet file source

2021-11-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36182. - Resolution: Fixed Issue resolved by pull request 34495 [https://github.com/apache/spark/pull/344

[jira] [Updated] (SPARK-36799) Pass queryExecution name in CLI

2021-11-10 Thread dzcxzl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dzcxzl updated SPARK-36799: --- Summary: Pass queryExecution name in CLI (was: Pass queryExecution name in CLI when only select query) > P

[jira] [Assigned] (SPARK-36799) Pass queryExecution name in CLI when only select query

2021-11-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36799: --- Assignee: dzcxzl > Pass queryExecution name in CLI when only select query > ---

[jira] [Resolved] (SPARK-36799) Pass queryExecution name in CLI when only select query

2021-11-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36799. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34041 [https://gith

[jira] [Commented] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442074#comment-17442074 ] Hyukjin Kwon commented on SPARK-37270: -- Hm, I can't reproduce this locally. Are you

[jira] [Commented] (SPARK-37278) Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442068#comment-17442068 ] Hyukjin Kwon commented on SPARK-37278: -- I am working on this. > Support YearMonthI

[jira] [Commented] (SPARK-37275) Support ANSI intervals in PySpark

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442066#comment-17442066 ] Hyukjin Kwon commented on SPARK-37275: -- cc [~maxgekk] FYI > Support ANSI intervals

[jira] [Updated] (SPARK-37281) Support DayTimeIntervalType in Py4J

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37281: - Description: This PR adds the support of YearMonthIntervalType in Py4J. For example, functions.l

[jira] [Created] (SPARK-37280) Support YearMonthIntervalType in Py4J

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37280: Summary: Support YearMonthIntervalType in Py4J Key: SPARK-37280 URL: https://issues.apache.org/jira/browse/SPARK-37280 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37281) Support DayTimeIntervalType in Py4J

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37281: Summary: Support DayTimeIntervalType in Py4J Key: SPARK-37281 URL: https://issues.apache.org/jira/browse/SPARK-37281 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37279) Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37279: Summary: Support DayTimeIntervalType in createDataFrame/toPandas and Python UDFs Key: SPARK-37279 URL: https://issues.apache.org/jira/browse/SPARK-37279 Project: Spar

[jira] [Updated] (SPARK-37278) Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37278: - Description: Implements the support of YearMonthIntervalType in: - Python UDFs - createDataFrame

[jira] [Created] (SPARK-37278) Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37278: Summary: Support YearMonthIntervalType in createDataFrame/toPandas and Python UDFs Key: SPARK-37278 URL: https://issues.apache.org/jira/browse/SPARK-37278 Project: Sp

[jira] [Created] (SPARK-37277) Support DayTimeIntervalType in Arrow

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37277: Summary: Support DayTimeIntervalType in Arrow Key: SPARK-37277 URL: https://issues.apache.org/jira/browse/SPARK-37277 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37276) Support YearMonthIntervalType in Arrow

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37276: Summary: Support YearMonthIntervalType in Arrow Key: SPARK-37276 URL: https://issues.apache.org/jira/browse/SPARK-37276 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37275) Support ANSI intervals in PySpark

2021-11-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-37275: Summary: Support ANSI intervals in PySpark Key: SPARK-37275 URL: https://issues.apache.org/jira/browse/SPARK-37275 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-37274) These parameters should be of type long, not int

2021-11-10 Thread hao (Jira)
hao created SPARK-37274: --- Summary: These parameters should be of type long, not int Key: SPARK-37274 URL: https://issues.apache.org/jira/browse/SPARK-37274 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37255) When Used with PyHive (by dropbox) query timeout doesn't result in propagation to the UI

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442058#comment-17442058 ] Hyukjin Kwon commented on SPARK-37255: -- That's very likely an issue in PyHive. > W

[jira] [Resolved] (SPARK-37255) When Used with PyHive (by dropbox) query timeout doesn't result in propagation to the UI

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37255. -- Resolution: Invalid > When Used with PyHive (by dropbox) query timeout doesn't result in > pr

[jira] [Commented] (SPARK-37273) Hidden File Metadata Support for Spark SQL

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442057#comment-17442057 ] Hyukjin Kwon commented on SPARK-37273: -- Don't we already have this in DSv2? e.g.) S

[jira] [Resolved] (SPARK-37273) Hidden File Metadata Support for Spark SQL

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37273. -- Resolution: Duplicate > Hidden File Metadata Support for Spark SQL > -

[jira] [Updated] (SPARK-37264) Exclude hadoop-client-api transitive dependency from orc-core

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37264: -- Summary: Exclude hadoop-client-api transitive dependency from orc-core (was: [SPARK-37264][BU

[jira] [Updated] (SPARK-37109) Install Java 17 on all of the Jenkins workers

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37109: -- Parent: (was: SPARK-33772) Issue Type: Bug (was: Sub-task) > Install Java 17 on a

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36900: - Assignee: Yang Jie > "SPARK-36464: size returns correct positive number even with over

[jira] [Closed] (SPARK-37109) Install Java 17 on all of the Jenkins workers

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-37109. - > Install Java 17 on all of the Jenkins workers > - > >

[jira] [Updated] (SPARK-37272) Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37272: -- Parent: SPARK-33772 Issue Type: Sub-task (was: Improvement) > Add `ExtendedRocksDBTes

[jira] [Updated] (SPARK-37272) Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37272: -- Description: Java 17 officially support Apple Silicon - JEP 391: macOS/AArch64 Port - https:/

[jira] [Updated] (SPARK-37272) Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37272: -- Description: Javava 17 officially support Apple Silicon - JEP 391: macOS/AArch64 Port - https

[jira] [Updated] (SPARK-37272) Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37272: -- Summary: Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon (was: Add Exten

[jira] [Resolved] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37272. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34547 [https://

[jira] [Assigned] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37272: - Assignee: Dongjoon Hyun > Add ExtendedRocksDBTest > --- > >

[jira] [Updated] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37270: - Labels: correctness (was: ) > Incorect result of filter using isNull condition > --

[jira] [Commented] (SPARK-37254) 100% CPU usage on Spark Thrift Server.

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442046#comment-17442046 ] Hyukjin Kwon commented on SPARK-37254: -- it would be much easier to investigate the

[jira] [Assigned] (SPARK-37233) Inline type hints for files in python/pyspark/mllib

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37233: Assignee: dch nguyen > Inline type hints for files in python/pyspark/mllib >

[jira] [Updated] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37260: - Fix Version/s: 3.2.1 > PYSPARK Arrow 3.2.0 docs link invalid > -

[jira] [Resolved] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37260. -- Resolution: Fixed > PYSPARK Arrow 3.2.0 docs link invalid > --

[jira] [Commented] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid

2021-11-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442044#comment-17442044 ] Hyukjin Kwon commented on SPARK-37260: -- oh yeah. that's fixed via #34475. There are

[jira] [Assigned] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37272: Assignee: Apache Spark > Add ExtendedRocksDBTest > --- > >

[jira] [Assigned] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37272: Assignee: (was: Apache Spark) > Add ExtendedRocksDBTest > --- > >

[jira] [Commented] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442020#comment-17442020 ] Apache Spark commented on SPARK-37272: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-37273) Hidden File Metadata Support for Spark SQL

2021-11-10 Thread Yaohua Zhao (Jira)
Yaohua Zhao created SPARK-37273: --- Summary: Hidden File Metadata Support for Spark SQL Key: SPARK-37273 URL: https://issues.apache.org/jira/browse/SPARK-37273 Project: Spark Issue Type: Improvem

[jira] [Created] (SPARK-37272) Add ExtendedRocksDBTest

2021-11-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37272: - Summary: Add ExtendedRocksDBTest Key: SPARK-37272 URL: https://issues.apache.org/jira/browse/SPARK-37272 Project: Spark Issue Type: Improvement C

[jira] [Comment Edited] (SPARK-33502) Large number of SELECT columns causes StackOverflowError

2021-11-10 Thread Arwin S Tio (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236434#comment-17236434 ] Arwin S Tio edited comment on SPARK-33502 at 11/10/21, 7:22 PM: --

[jira] [Resolved] (SPARK-35557) Adapt uses of JDK 17 Internal APIs

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35557. --- Resolution: Duplicate This is superseded by SPARK-36796 via adding `--add-open` options. >

[jira] [Resolved] (SPARK-37265) Support Java 17 in `dev/test-dependencies.sh`

2021-11-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37265. --- Resolution: Invalid Let me close this Invalid. > Support Java 17 in `dev/test-dependencies.

[jira] [Resolved] (SPARK-37271) Spark OOM issue

2021-11-10 Thread M Shadab (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M Shadab resolved SPARK-37271. -- Resolution: Fixed done > Spark OOM issue > --- > > Key: SPARK-37271 >

[jira] [Commented] (SPARK-37271) Spark OOM issue

2021-11-10 Thread M Shadab (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441805#comment-17441805 ] M Shadab commented on SPARK-37271: -- Memory increased for the container > Spark OOM iss

[jira] [Updated] (SPARK-37271) Spark OOM issue

2021-11-10 Thread M Shadab (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M Shadab updated SPARK-37271: - Shepherd: M Shadab > Spark OOM issue > --- > > Key: SPARK-37271 >

[jira] [Created] (SPARK-37271) Spark OOM issue

2021-11-10 Thread M Shadab (Jira)
M Shadab created SPARK-37271: Summary: Spark OOM issue Key: SPARK-37271 URL: https://issues.apache.org/jira/browse/SPARK-37271 Project: Spark Issue Type: Bug Components: Spark Submit

[jira] [Commented] (SPARK-36575) Executor lost may cause spark stage to hang

2021-11-10 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441796#comment-17441796 ] wuyi commented on SPARK-36575: -- FYI: the fix is reverted due to test issues. > Executor lo

[jira] [Assigned] (SPARK-37045) Unify v1 and v2 ALTER TABLE .. ADD COLUMNS tests

2021-11-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37045: Assignee: Max Gekk > Unify v1 and v2 ALTER TABLE .. ADD COLUMNS tests > -

[jira] [Commented] (SPARK-37045) Unify v1 and v2 ALTER TABLE .. ADD COLUMNS tests

2021-11-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441755#comment-17441755 ] Max Gekk commented on SPARK-37045: -- I am working on this. > Unify v1 and v2 ALTER TABL

[jira] [Resolved] (SPARK-37236) Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/

2021-11-10 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37236. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34510

[jira] [Assigned] (SPARK-37236) Inline type hints for KernelDensity.pyi, test.py in python/pyspark/mllib/stat/

2021-11-10 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37236: -- Assignee: dch nguyen > Inline type hints for KernelDensity.pyi, test.py in py

[jira] [Updated] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-10 Thread Tomasz Kus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Kus updated SPARK-37270: --- Component/s: SQL > Incorect result of filter using isNull condition > --

[jira] [Resolved] (SPARK-37261) Check adding partitions with ANSI intervals

2021-11-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37261. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34537 [https://github.com

[jira] [Created] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-10 Thread Tomasz Kus (Jira)
Tomasz Kus created SPARK-37270: -- Summary: Incorect result of filter using isNull condition Key: SPARK-37270 URL: https://issues.apache.org/jira/browse/SPARK-37270 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-37269) The partitionOverwriteMode option is not respected when using insertInto

2021-11-10 Thread David Szakallas (Jira)
David Szakallas created SPARK-37269: --- Summary: The partitionOverwriteMode option is not respected when using insertInto Key: SPARK-37269 URL: https://issues.apache.org/jira/browse/SPARK-37269 Projec

[jira] [Commented] (SPARK-37268) Remove unused method call in FileScanRDD

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441669#comment-17441669 ] Apache Spark commented on SPARK-37268: -- User 'zuston' has created a pull request fo

[jira] [Assigned] (SPARK-37268) Remove unused method call in FileScanRDD

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37268: Assignee: (was: Apache Spark) > Remove unused method call in FileScanRDD > --

[jira] [Assigned] (SPARK-37268) Remove unused method call in FileScanRDD

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37268: Assignee: Apache Spark > Remove unused method call in FileScanRDD > -

[jira] [Created] (SPARK-37268) Remove unused method call in FileScanRDD

2021-11-10 Thread Junfan Zhang (Jira)
Junfan Zhang created SPARK-37268: Summary: Remove unused method call in FileScanRDD Key: SPARK-37268 URL: https://issues.apache.org/jira/browse/SPARK-37268 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-37022) Use black as a formatter for the whole PySpark codebase.

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441612#comment-17441612 ] Apache Spark commented on SPARK-37022: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-37022) Use black as a formatter for the whole PySpark codebase.

2021-11-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441610#comment-17441610 ] Apache Spark commented on SPARK-37022: -- User 'HyukjinKwon' has created a pull reque

  1   2   >