[jira] [Assigned] (SPARK-36998) Handle concurrent eviction of same application in SHS

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36998: Assignee: Apache Spark > Handle concurrent eviction of same application in SHS >

[jira] [Assigned] (SPARK-36998) Handle concurrent eviction of same application in SHS

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36998: Assignee: (was: Apache Spark) > Handle concurrent eviction of same application in

[jira] [Reopened] (SPARK-36998) Handle concurrent eviction of same application in SHS

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-36998: -- Assignee: (was: Thejdeep Gudivada) Reverted in

[jira] [Updated] (SPARK-36998) Handle concurrent eviction of same application in SHS

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36998: - Fix Version/s: (was: 3.3.0) > Handle concurrent eviction of same application in SHS >

[jira] [Resolved] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37198. -- Resolution: Invalid > pyspark.pandas read_csv() and to_csv() should handle local files >

[jira] [Commented] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440920#comment-17440920 ] Hyukjin Kwon commented on SPARK-37198: -- I think this is Databricks' specific issue. Let's loop with

[jira] [Resolved] (SPARK-37245) SAP HANA : Truncate doesn't truncate the table but rather drops the table

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37245. -- Resolution: Invalid > SAP HANA : Truncate doesn't truncate the table but rather drops the

[jira] [Commented] (SPARK-37245) SAP HANA : Truncate doesn't truncate the table but rather drops the table

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440919#comment-17440919 ] Hyukjin Kwon commented on SPARK-37245: -- Assuming from the stacktrace, it seems like the plugin's

[jira] [Resolved] (SPARK-37242) option -v is broken?

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37242. -- Resolution: Duplicate > option -v is broken? > > > Key:

[jira] [Commented] (SPARK-37202) Temp view didn't collect temp function that registered with catalog API

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440912#comment-17440912 ] Apache Spark commented on SPARK-37202: -- User 'linhongliu-db' has created a pull request for this

[jira] [Commented] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440913#comment-17440913 ] Apache Spark commented on SPARK-37244: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-08 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440895#comment-17440895 ] melin commented on SPARK-24907: --- SaveMode.Overwrite,  Support for the mysql replace into tablename syntax. 

[jira] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-08 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907 ] melin deleted comment on SPARK-24907: --- was (Author: melin): SaveMode > Migrate JDBC data source to DataSource API v2 > - > > Key:

[jira] [Commented] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-08 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440894#comment-17440894 ] melin commented on SPARK-24907: --- SaveMode > Migrate JDBC data source to DataSource API v2 >

[jira] [Commented] (SPARK-24907) Migrate JDBC data source to DataSource API v2

2021-11-08 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440893#comment-17440893 ] melin commented on SPARK-24907: --- add presql and postsql  presql: Run the presql before the task start

[jira] [Resolved] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37244. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34526

[jira] [Resolved] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-37239. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34520

[jira] [Assigned] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-37239: Assignee: Yang Jie > Avoid unnecessary `setReplication` in Yarn mode >

[jira] [Resolved] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37246. -- Resolution: Duplicate > Support Python 3.10 in PySpark > -- > >

[jira] [Commented] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440872#comment-17440872 ] Xinrong Meng commented on SPARK-37246: -- Oh sorry for the confusion. I was about to resolve the

[jira] [Commented] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440865#comment-17440865 ] Dongjoon Hyun commented on SPARK-37246: --- Hi, may I ask the reason why you filed JIRAs and marked

[jira] [Resolved] (SPARK-37249) ImportError: cannot import name 'Callable' from 'collections'

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37249. --- Resolution: Duplicate > ImportError: cannot import name 'Callable' from 'collections' >

[jira] [Closed] (SPARK-37249) ImportError: cannot import name 'Callable' from 'collections'

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-37249. - > ImportError: cannot import name 'Callable' from 'collections' >

[jira] [Resolved] (SPARK-37252) Ignore test_memory_limit on non-Linux environment

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37252. --- Fix Version/s: 3.1.3 3.2.1 3.3.0 Assignee:

[jira] [Resolved] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37247. -- Resolution: Invalid > Failed test_create_nan_decimal_dataframe >

[jira] [Resolved] (SPARK-37250) Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37250. -- Resolution: Invalid > Failed test_capture_user_friendly_exception >

[jira] [Resolved] (SPARK-37248) Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37248. -- Resolution: Invalid > Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

[jira] [Resolved] (SPARK-37251) Failed _joinAsOf doctest

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37251. -- Resolution: Invalid > Failed _joinAsOf doctest > > >

[jira] [Commented] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440859#comment-17440859 ] Yang Jie commented on SPARK-37239: -- Haha, I give a PR yesterday:)   > Avoid unnecessary

[jira] [Commented] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Victor Wong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440852#comment-17440852 ] Victor Wong commented on SPARK-37239: - HDFS `setReplication` is a time-consuming rpc call, it should

[jira] [Commented] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Victor Wong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440851#comment-17440851 ] Victor Wong commented on SPARK-37239: - [~LuciferYang] It seems the reporter could come up with a PR

[jira] [Assigned] (SPARK-37252) Ignore test_memory_limit on non-Linux environment

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37252: Assignee: (was: Apache Spark) > Ignore test_memory_limit on non-Linux environment >

[jira] [Assigned] (SPARK-37252) Ignore test_memory_limit on non-Linux environment

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37252: Assignee: Apache Spark > Ignore test_memory_limit on non-Linux environment >

[jira] [Commented] (SPARK-37252) Ignore test_memory_limit on non-Linux environment

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440848#comment-17440848 ] Apache Spark commented on SPARK-37252: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Resolved] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37235. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34515

[jira] [Assigned] (SPARK-37235) Inline type hints for distribution.py and __init__.py in python/pyspark/mllib/stat

2021-11-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37235: Assignee: dch nguyen > Inline type hints for distribution.py and __init__.py in >

[jira] [Created] (SPARK-37252) Ignore test_memory_limit on non-Linux environment

2021-11-08 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37252: - Summary: Ignore test_memory_limit on non-Linux environment Key: SPARK-37252 URL: https://issues.apache.org/jira/browse/SPARK-37252 Project: Spark Issue

[jira] [Commented] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440843#comment-17440843 ] Dongjoon Hyun commented on SPARK-37246: --- Thank you for leading this, [~XinrongM]. > Support

[jira] [Created] (SPARK-37251) Failed _joinAsOf doctest

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37251: Summary: Failed _joinAsOf doctest Key: SPARK-37251 URL: https://issues.apache.org/jira/browse/SPARK-37251 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37250) Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37250: Summary: Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests) Key: SPARK-37250 URL: https://issues.apache.org/jira/browse/SPARK-37250

[jira] [Created] (SPARK-37249) ImportError: cannot import name 'Callable' from 'collections'

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37249: Summary: ImportError: cannot import name 'Callable' from 'collections' Key: SPARK-37249 URL: https://issues.apache.org/jira/browse/SPARK-37249 Project: Spark

[jira] [Created] (SPARK-37248) Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37248: Summary: Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests) Key: SPARK-37248 URL: https://issues.apache.org/jira/browse/SPARK-37248 Project:

[jira] [Updated] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37247: - Description: {code:java}   File

[jira] [Created] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37247: Summary: Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests) Key: SPARK-37247 URL: https://issues.apache.org/jira/browse/SPARK-37247

[jira] [Updated] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37246: - Issue Type: Umbrella (was: Improvement) > Support Python 3.10 in PySpark >

[jira] [Updated] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37246: - Description: Though pyarrow doesn’t support python 3.10 yet

[jira] [Created] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37246: Summary: Support Python 3.10 in PySpark Key: SPARK-37246 URL: https://issues.apache.org/jira/browse/SPARK-37246 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-35867) Enable vectorized read for VectorizedPlainValuesReader.readBooleans

2021-11-08 Thread Kazuyuki Tanimura (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440813#comment-17440813 ] Kazuyuki Tanimura commented on SPARK-35867: --- I am working on this > Enable vectorized read

[jira] [Created] (SPARK-37245) SAP HANA : Truncate doesn't truncate the table but rather drops the table

2021-11-08 Thread Balaji Balasubramaniam (Jira)
Balaji Balasubramaniam created SPARK-37245: -- Summary: SAP HANA : Truncate doesn't truncate the table but rather drops the table Key: SPARK-37245 URL: https://issues.apache.org/jira/browse/SPARK-37245

[jira] [Assigned] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37244: Assignee: Apache Spark (was: Dongjoon Hyun) > Build and test on Python 3.10 >

[jira] [Assigned] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37244: Assignee: Dongjoon Hyun (was: Apache Spark) > Build and test on Python 3.10 >

[jira] [Commented] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440757#comment-17440757 ] Apache Spark commented on SPARK-37244: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37244: - Assignee: Dongjoon Hyun > Build and test on Python 3.10 >

[jira] [Updated] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37244: -- Description: Python 3.10 introduced breaking changes. We need to update PySpark. -

[jira] [Updated] (SPARK-37244) Build and test on Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37244: -- Summary: Build and test on Python 3.10 (was: Build and test with Python 3.10) > Build and

[jira] [Updated] (SPARK-37244) Build and test with Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37244: -- Summary: Build and test with Python 3.10 (was: Support Python 3.10) > Build and test with

[jira] [Updated] (SPARK-37244) Support Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37244: -- Target Version/s: 3.3.0 > Support Python 3.10 > --- > > Key:

[jira] [Updated] (SPARK-37244) Support Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37244: -- Description: Python 3.10 introduced breaking changes. We need to update PySpark. -

[jira] [Created] (SPARK-37244) Support Python 3.10

2021-11-08 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37244: - Summary: Support Python 3.10 Key: SPARK-37244 URL: https://issues.apache.org/jira/browse/SPARK-37244 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37120) Add Java17 GitHub Action build and test job

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440745#comment-17440745 ] Apache Spark commented on SPARK-37120: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-37120) Add Java17 GitHub Action build and test job

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440744#comment-17440744 ] Apache Spark commented on SPARK-37120: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37196: - Assignee: angerszhu (was: angerszhu) > NPE in >

[jira] [Assigned] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37196: - Assignee: Yang Jie > NPE in >

[jira] [Resolved] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37196. --- Fix Version/s: 3.3.0 3.0.4 3.2.1

[jira] [Assigned] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37196: - Assignee: angerszhu (was: Yang Jie) > NPE in >

[jira] [Commented] (SPARK-26413) SPIP: RDD Arrow Support in Spark Core and PySpark

2021-11-08 Thread Ruslan Dautkhanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440731#comment-17440731 ] Ruslan Dautkhanov commented on SPARK-26413: --- [https://github.com/apache/spark/pull/34505] is

[jira] (SPARK-36895) Add Create Index syntax support

2021-11-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895 ] Kousuke Saruta deleted comment on SPARK-36895: was (Author: sarutak): The change in https://github.com/apache/spark/pull/34148 was reverted and resolved again in

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

2021-11-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440696#comment-17440696 ] Kousuke Saruta commented on SPARK-36895: The change in

[jira] [Updated] (SPARK-37217) The number of dynamic partitions should early check when writing to external tables

2021-11-08 Thread dzcxzl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dzcxzl updated SPARK-37217: --- Summary: The number of dynamic partitions should early check when writing to external tables (was: Dynamic

[jira] [Resolved] (SPARK-37237) Upgrade kubernetes-client to 5.9.0

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37237. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34511

[jira] [Assigned] (SPARK-37237) Upgrade kubernetes-client to 5.9.0

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37237: - Assignee: Dongjoon Hyun > Upgrade kubernetes-client to 5.9.0 >

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440570#comment-17440570 ] Apache Spark commented on SPARK-36895: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440569#comment-17440569 ] Apache Spark commented on SPARK-36895: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37240) Cannot read partitioned parquet files with ANSI interval partition values

2021-11-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-37240. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in

[jira] [Assigned] (SPARK-37241) Upgrade Jackson to 2.13.0

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37241: - Assignee: Dongjoon Hyun > Upgrade Jackson to 2.13.0 > - > >

[jira] [Resolved] (SPARK-37241) Upgrade Jackson to 2.13.0

2021-11-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37241. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34516

[jira] [Resolved] (SPARK-37225) Read/write dataframes with ANSI intervals from/to Avro files

2021-11-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37225. - Resolution: Fixed Issue resolved by pull request 34503

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440517#comment-17440517 ] Apache Spark commented on SPARK-36895: -- User 'sarutak' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37243. -- Fix Version/s: 3.3.0 Resolution: Fixed Resolved by

[jira] [Assigned] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37243: Assignee: yikf > Fix the format of the document > -- > >

[jira] [Commented] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440416#comment-17440416 ] Apache Spark commented on SPARK-37243: -- User 'yikf' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37243: Assignee: Apache Spark > Fix the format of the document > --

[jira] [Assigned] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37243: Assignee: (was: Apache Spark) > Fix the format of the document >

[jira] [Commented] (SPARK-37243) Fix the format of the document

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440414#comment-17440414 ] Apache Spark commented on SPARK-37243: -- User 'yikf' has created a pull request for this issue:

[jira] [Updated] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37243: - Description: Currently, the format of `sql-ref-syntax-dml-insert-table.md` is: [currently

[jira] [Assigned] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37239: Assignee: (was: Apache Spark) > Avoid unnecessary `setReplication` in Yarn mode >

[jira] [Commented] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440413#comment-17440413 ] Apache Spark commented on SPARK-37239: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-37239) Avoid unnecessary `setReplication` in Yarn mode

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37239: Assignee: Apache Spark > Avoid unnecessary `setReplication` in Yarn mode >

[jira] [Updated] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37243: - Attachment: currently format.png > Fix the format of the document > -- > >

[jira] [Updated] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37243: - Description: Currently, the format of `sql-ref-syntax-dml-insert-overwrite-table.md` is: [currently

[jira] [Updated] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37243: - Description: Currently, the format of `sql-ref-syntax-dml-insert-overwrite-table.md` is: [currently

[jira] [Updated] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yikf updated SPARK-37243: - Description: Currently, the format of `sql-ref-syntax-dml-insert-overwrite-table.md` is:   it should be

[jira] [Created] (SPARK-37243) Fix the format of the document

2021-11-08 Thread yikf (Jira)
yikf created SPARK-37243: Summary: Fix the format of the document Key: SPARK-37243 URL: https://issues.apache.org/jira/browse/SPARK-37243 Project: Spark Issue Type: Improvement Components:

[jira] [Commented] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440401#comment-17440401 ] Apache Spark commented on SPARK-37196: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37196: Assignee: Apache Spark > NPE in >

[jira] [Assigned] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37196: Assignee: (was: Apache Spark) > NPE in >

[jira] [Commented] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440402#comment-17440402 ] Apache Spark commented on SPARK-37196: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-37196) NPE in org.apache.spark.sql.hive.HiveShim$.toCatalystDecimal(HiveShim.scala:106)

2021-11-08 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440397#comment-17440397 ] angerszhu commented on SPARK-37196: --- It's caused by you create a df with   {code} root |-- name:

[jira] [Updated] (SPARK-37242) option -v is broken?

2021-11-08 Thread Patrice DUROUX (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrice DUROUX updated SPARK-37242: --- Description: Hi, Here is the output:   {{$ spark-shell -v}} {{*[CUT]*}} {{Parsed

[jira] [Created] (SPARK-37242) option -v is broken?

2021-11-08 Thread Patrice DUROUX (Jira)
Patrice DUROUX created SPARK-37242: -- Summary: option -v is broken? Key: SPARK-37242 URL: https://issues.apache.org/jira/browse/SPARK-37242 Project: Spark Issue Type: Improvement

  1   2   >