[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015) In folder A having two

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Commented] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428007#comment-17428007 ] Apache Spark commented on SPARK-36993: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36993: Assignee: Apache Spark > Fix json_tupe throw NPE if fields exist no foldable null value

[jira] [Assigned] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36993: Assignee: (was: Apache Spark) > Fix json_tupe throw NPE if fields exist no foldable

[jira] [Commented] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428006#comment-17428006 ] Apache Spark commented on SPARK-36993: -- User 'ulysses-you' has created a pull request for this

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Affects Version/s: 3.0.3 > Fix json_tupe throw NPE if fields exist no foldable null value >

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null field

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Summary: Fix json_tupe throw NPE if fields exist no foldable null field (was: Fix json_tupe throw

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Summary: Fix json_tupe throw NPE if fields exist no foldable null value (was: Fix json_tupe throw

[jira] [Created] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null column

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36993: - Summary: Fix json_tupe throw NPE if fields exist no foldable null column Key: SPARK-36993 URL: https://issues.apache.org/jira/browse/SPARK-36993 Project: Spark

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null column

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Description: If json_tuple exists no foldable null field, Spark would throw NPE during eval

[jira] [Resolved] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36953. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34219

[jira] [Assigned] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36953: Assignee: Hyukjin Kwon > Expose SQL state and error class in PySpark exceptions >

[jira] [Resolved] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36794. - Resolution: Fixed Issue resolved by pull request 34247

[jira] [Updated] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36794: Summary: Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: Apache Spark > "SPARK-36464: size returns correct positive number even with

[jira] [Resolved] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

2021-10-12 Thread huangtengfei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei resolved SPARK-36954. -- Resolution: Not A Problem > Fast fail with explicit err msg when calling withWatermark on

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: (was: Apache Spark) > "SPARK-36464: size returns correct positive number

[jira] [Reopened] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-36900: -- Assignee: (was: Sean R. Owen) > "SPARK-36464: size returns correct positive number even

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427992#comment-17427992 ] Hyukjin Kwon commented on SPARK-36900: -- Reverted in:

[jira] [Updated] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36900: - Fix Version/s: (was: 3.2.1) (was: 3.3.0) > "SPARK-36464: size

[jira] [Commented] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427981#comment-17427981 ] Apache Spark commented on SPARK-36992: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: (was: Apache Spark) > Improve byte array sort perf by unify getPrefix

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: Apache Spark > Improve byte array sort perf by unify getPrefix function of

[jira] [Created] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36992: - Summary: Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray Key: SPARK-36992 URL: https://issues.apache.org/jira/browse/SPARK-36992

[jira] [Commented] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427976#comment-17427976 ] Hyukjin Kwon commented on SPARK-36971: -- I suggest you do contact AWS or Databricks to follow up the

[jira] [Resolved] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36971. -- Resolution: Invalid > Query files directly with SQL is broken (with Glue) >

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them

[jira] [Resolved] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36985. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Attachment: file2.parquet file1.parquet > ignoreCorruptFiles does not work when schema change

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427970#comment-17427970 ] Hyukjin Kwon commented on SPARK-36989: -- Adding mypy tests would be super awesome! > Migrate type

[jira] [Assigned] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36961: Assignee: Takuya Ueshin > Use PEP526 style variable type hints >

[jira] [Resolved] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36961. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34227

[jira] [Resolved] (SPARK-36981) Upgrade joda-time to 2.10.12

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36981. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427966#comment-17427966 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: (was: Apache Spark) > Future typing errors in pyspark.pandas >

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: Apache Spark > Future typing errors in pyspark.pandas >

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427967#comment-17427967 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427958#comment-17427958 ] Apache Spark commented on SPARK-23626: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36979: - Assignee: XiDuo You > Add RewriteLateralSubquery rule into nonExcludableRules >

[jira] [Updated] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36979: -- Issue Type: Bug (was: Improvement) > Add RewriteLateralSubquery rule into nonExcludableRules

[jira] [Resolved] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36979. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34260

[jira] [Commented] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427922#comment-17427922 ] Xinrong Meng commented on SPARK-36991: -- I am working on this. > Inline type hints for

[jira] [Created] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36991: Summary: Inline type hints for spark/python/pyspark/sql/streaming.py Key: SPARK-36991 URL: https://issues.apache.org/jira/browse/SPARK-36991 Project: Spark

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Description: The code below does not work on both Spark 3.1 and Spark 3.2. Part of the issue

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Environment: (was: Python repro: {code:java} import os from pyspark.sql.functions import *

[jira] [Resolved] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

2021-10-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36951. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Created] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
Catalin Toda created SPARK-36990: Summary: Long columns cannot read columns with INT32 type in the parquet file Key: SPARK-36990 URL: https://issues.apache.org/jira/browse/SPARK-36990 Project: Spark

[jira] [Updated] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-36989: --- Description: Before the migration, {{pyspark-stubs}} contained a set of [data

[jira] [Comment Edited] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427889#comment-17427889 ] Maciej Szymkiewicz edited comment on SPARK-36989 at 10/12/21, 7:23 PM:

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427889#comment-17427889 ] Maciej Szymkiewicz commented on SPARK-36989: FYI [~hyukjin.kwon] [~XinrongM] [~ueshin] >

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427888#comment-17427888 ] Maciej Szymkiewicz commented on SPARK-36989: Currently I am working on [some

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: (was: Apache Spark) > Allow Spark on Kube to operate without polling or

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: Apache Spark > Allow Spark on Kube to operate without polling or watchers >

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427886#comment-17427886 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request for this issue:

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427884#comment-17427884 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request for this issue:

[jira] [Created] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-36989: -- Summary: Migrate type hint data tests Key: SPARK-36989 URL: https://issues.apache.org/jira/browse/SPARK-36989 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427852#comment-17427852 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: Apache Spark > InferConstraints rule should create IsNotNull constraints on

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: (was: Apache Spark) > InferConstraints rule should create IsNotNull

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427851#comment-17427851 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request for this issue:

[jira] [Updated] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Utkarsh Agarwal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Utkarsh Agarwal updated SPARK-36978: Description:

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

2021-10-12 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427825#comment-17427825 ] Shardul Mahadik commented on SPARK-36877: - {quote} Getting RDD means the physical plan is

[jira] [Resolved] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36970. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34237

[jira] [Assigned] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-36970: Assignee: Yang Jie > Manual disabled format `B` for `date_format` function to compatibility with

[jira] [Updated] (SPARK-36988) What ciphers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Summary: What ciphers spark support for internode communication? (was: What chipers spark support for

[jira] [Assigned] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36867: --- Assignee: Wenchen Fan > Misleading Error Message with Invalid Column and Group By >

[jira] [Resolved] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36867. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34244

[jira] [Resolved] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36914. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34236

[jira] [Assigned] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36914: --- Assignee: Huaxin Gao > Implement dropIndex and listIndexes in JDBC (MySQL dialect) >

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Environment: (was: {{Spark documentation mention this:}}

[jira] [Created] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
zoli created SPARK-36988: Summary: What chipers spark support for internode communication? Key: SPARK-36988 URL: https://issues.apache.org/jira/browse/SPARK-36988 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string (was: ignoreCorruptFiles does

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: (was: Apache Spark) > Add Doc about FROM statement >

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427672#comment-17427672 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: Apache Spark > Add Doc about FROM statement > > >

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427673#comment-17427673 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull request for this

[jira] [Created] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread angerszhu (Jira)
angerszhu created SPARK-36987: - Summary: Add Doc about FROM statement Key: SPARK-36987 URL: https://issues.apache.org/jira/browse/SPARK-36987 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-36972) Add max_by/min_by API to PySpark

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36972. Fix Version/s: 3.3.0 Assignee: Leona Yoda Resolution: Fixed Issue

[jira] [Created] (SPARK-36986) Improving external schema management flexibility

2021-10-12 Thread Rodrigo Boavida (Jira)
Rodrigo Boavida created SPARK-36986: --- Summary: Improving external schema management flexibility Key: SPARK-36986 URL: https://issues.apache.org/jira/browse/SPARK-36986 Project: Spark Issue

[jira] [Commented] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427627#comment-17427627 ] Apache Spark commented on SPARK-36979: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36949: Assignee: (was: Apache Spark) > Fix CREATE TABLE AS SELECT of ANSI intervals >

[jira] [Assigned] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36949: Assignee: Apache Spark > Fix CREATE TABLE AS SELECT of ANSI intervals >

[jira] [Commented] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427624#comment-17427624 ] Apache Spark commented on SPARK-36949: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427618#comment-17427618 ] Apache Spark commented on SPARK-36976: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36976: Assignee: (was: Apache Spark) > Add max_by/min_by API to SparkR >

[jira] [Commented] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427616#comment-17427616 ] Apache Spark commented on SPARK-36976: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36976: Assignee: Apache Spark > Add max_by/min_by API to SparkR >

[jira] [Comment Edited] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

2021-10-12 Thread Kanishka Chauhan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427584#comment-17427584 ] Kanishka Chauhan edited comment on SPARK-24156 at 10/12/21, 10:41 AM:

[jira] [Commented] (SPARK-36921) The DIV function should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427611#comment-17427611 ] Apache Spark commented on SPARK-36921: -- User 'Peng-Lei' has created a pull request for this issue:

  1   2   >