[jira] [Resolved] (SPARK-36973) Deduplicate prepare data method for HistogramPlotBase and KdePlotBase

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36973. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34251 [https://gi

[jira] [Assigned] (SPARK-36973) Deduplicate prepare data method for HistogramPlotBase and KdePlotBase

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36973: Assignee: dch nguyen > Deduplicate prepare data method for HistogramPlotBase and KdePlotB

[jira] [Created] (SPARK-36994) Upgrade Apache Thrift

2021-10-12 Thread kaja girish (Jira)
kaja girish created SPARK-36994: --- Summary: Upgrade Apache Thrift Key: SPARK-36994 URL: https://issues.apache.org/jira/browse/SPARK-36994 Project: Spark Issue Type: Bug Components: Sec

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015) In folder A having two pa

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015) In folder A having two pa

[jira] [Commented] (SPARK-36972) Add max_by/min_by API to PySpark

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428036#comment-17428036 ] Apache Spark commented on SPARK-36972: -- User 'yoda-mon' has created a pull request

[jira] [Resolved] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36976. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34258 [https://gi

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Attachment: (was: file1.parquet) > ignoreCorruptFiles does not work when schema change from int to string wh

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Attachment: (was: file2.parquet) > ignoreCorruptFiles does not work when schema change from int to string wh

[jira] [Assigned] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36976: Assignee: Leona Yoda > Add max_by/min_by API to SparkR > ---

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015) In folder A having two pa

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than X records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string when a file having more than X r

[jira] [Updated] (SPARK-36993) Fix json_tuple throw NPE if fields exist no foldable null value

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36993: --- Summary: Fix json_tuple throw NPE if fields exist no foldable null value (was: Fix json_tup

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015) In folder A having two pa

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string when a file having more than 35 records

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Commented] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428007#comment-17428007 ] Apache Spark commented on SPARK-36993: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36993: Assignee: Apache Spark > Fix json_tupe throw NPE if fields exist no foldable null value >

[jira] [Assigned] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36993: Assignee: (was: Apache Spark) > Fix json_tupe throw NPE if fields exist no foldable n

[jira] [Commented] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428006#comment-17428006 ] Apache Spark commented on SPARK-36993: -- User 'ulysses-you' has created a pull reque

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Affects Version/s: 3.0.3 > Fix json_tupe throw NPE if fields exist no foldable null value > --

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null field

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Summary: Fix json_tupe throw NPE if fields exist no foldable null field (was: Fix json_tupe throw NPE

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null value

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Summary: Fix json_tupe throw NPE if fields exist no foldable null value (was: Fix json_tupe throw NPE

[jira] [Created] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null column

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36993: - Summary: Fix json_tupe throw NPE if fields exist no foldable null column Key: SPARK-36993 URL: https://issues.apache.org/jira/browse/SPARK-36993 Project: Spark Is

[jira] [Updated] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null column

2021-10-12 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-36993: -- Description: If json_tuple exists no foldable null field, Spark would throw NPE during eval field.toS

[jira] [Resolved] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36953. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34219 [https://gi

[jira] [Assigned] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36953: Assignee: Hyukjin Kwon > Expose SQL state and error class in PySpark exceptions > ---

[jira] [Resolved] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36794. - Resolution: Fixed Issue resolved by pull request 34247 [https://github.com/apache/spark/pull/342

[jira] [Updated] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36794: Summary: Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join (was:

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: Apache Spark > "SPARK-36464: size returns correct positive number even with ove

[jira] [Resolved] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

2021-10-12 Thread huangtengfei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei resolved SPARK-36954. -- Resolution: Not A Problem > Fast fail with explicit err msg when calling withWatermark on non-

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: (was: Apache Spark) > "SPARK-36464: size returns correct positive number ev

[jira] [Reopened] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-36900: -- Assignee: (was: Sean R. Owen) > "SPARK-36464: size returns correct positive number even

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427992#comment-17427992 ] Hyukjin Kwon commented on SPARK-36900: -- Reverted in: https://github.com/apache/spa

[jira] [Updated] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36900: - Fix Version/s: (was: 3.2.1) (was: 3.3.0) > "SPARK-36464: size returns

[jira] [Commented] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427981#comment-17427981 ] Apache Spark commented on SPARK-36992: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: (was: Apache Spark) > Improve byte array sort perf by unify getPrefix funct

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: Apache Spark > Improve byte array sort perf by unify getPrefix function of UTF8

[jira] [Created] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36992: - Summary: Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray Key: SPARK-36992 URL: https://issues.apache.org/jira/browse/SPARK-36992 Projec

[jira] [Commented] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427976#comment-17427976 ] Hyukjin Kwon commented on SPARK-36971: -- I suggest you do contact AWS or Databricks

[jira] [Resolved] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36971. -- Resolution: Invalid > Query files directly with SQL is broken (with Glue) > --

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Resolved] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36985. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in https:

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Attachment: file2.parquet file1.parquet > ignoreCorruptFiles does not work when schema change fr

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427970#comment-17427970 ] Hyukjin Kwon commented on SPARK-36989: -- Adding mypy tests would be super awesome!

[jira] [Assigned] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36961: Assignee: Takuya Ueshin > Use PEP526 style variable type hints >

[jira] [Resolved] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36961. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34227 [https://gi

[jira] [Resolved] (SPARK-36981) Upgrade joda-time to 2.10.12

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36981. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in https://github.com/apache/

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427966#comment-17427966 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: (was: Apache Spark) > Future typing errors in pyspark.pandas >

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: Apache Spark > Future typing errors in pyspark.pandas > ---

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427967#comment-17427967 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427958#comment-17427958 ] Apache Spark commented on SPARK-23626: -- User 'JoshRosen' has created a pull request

[jira] [Assigned] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36979: - Assignee: XiDuo You > Add RewriteLateralSubquery rule into nonExcludableRules > ---

[jira] [Updated] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36979: -- Issue Type: Bug (was: Improvement) > Add RewriteLateralSubquery rule into nonExcludableRules

[jira] [Resolved] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36979. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34260 [https://

[jira] [Commented] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427922#comment-17427922 ] Xinrong Meng commented on SPARK-36991: -- I am working on this. > Inline type hints

[jira] [Created] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36991: Summary: Inline type hints for spark/python/pyspark/sql/streaming.py Key: SPARK-36991 URL: https://issues.apache.org/jira/browse/SPARK-36991 Project: Spark

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Description: The code below does not work on both Spark 3.1 and Spark 3.2. Part of the issue is

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Environment: (was: Python repro: {code:java} import os from pyspark.sql.functions import * fr

[jira] [Resolved] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

2021-10-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36951. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue resolved

[jira] [Created] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
Catalin Toda created SPARK-36990: Summary: Long columns cannot read columns with INT32 type in the parquet file Key: SPARK-36990 URL: https://issues.apache.org/jira/browse/SPARK-36990 Project: Spark

[jira] [Updated] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-36989: --- Description: Before the migration, {{pyspark-stubs}} contained a set of [data tests

[jira] [Comment Edited] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427889#comment-17427889 ] Maciej Szymkiewicz edited comment on SPARK-36989 at 10/12/21, 7:23 PM: ---

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427889#comment-17427889 ] Maciej Szymkiewicz commented on SPARK-36989: FYI [~hyukjin.kwon] [~XinrongM]

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427888#comment-17427888 ] Maciej Szymkiewicz commented on SPARK-36989: Currently I am working on [some

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: (was: Apache Spark) > Allow Spark on Kube to operate without polling or wat

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: Apache Spark > Allow Spark on Kube to operate without polling or watchers > ---

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427886#comment-17427886 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request f

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427884#comment-17427884 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request f

[jira] [Created] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-36989: -- Summary: Migrate type hint data tests Key: SPARK-36989 URL: https://issues.apache.org/jira/browse/SPARK-36989 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427852#comment-17427852 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: Apache Spark > InferConstraints rule should create IsNotNull constraints on the

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: (was: Apache Spark) > InferConstraints rule should create IsNotNull constra

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427851#comment-17427851 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request

[jira] [Updated] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Utkarsh Agarwal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Utkarsh Agarwal updated SPARK-36978: Description: [InferFiltersFromConstraints|https://github.com/apache/spark/blob/05c0fa57388

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

2021-10-12 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427825#comment-17427825 ] Shardul Mahadik commented on SPARK-36877: - {quote} Getting RDD means the physica

[jira] [Resolved] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36970. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34237 [https://github.com

[jira] [Assigned] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-36970: Assignee: Yang Jie > Manual disabled format `B` for `date_format` function to compatibility with

[jira] [Updated] (SPARK-36988) What ciphers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Summary: What ciphers spark support for internode communication? (was: What chipers spark support for internode

[jira] [Assigned] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36867: --- Assignee: Wenchen Fan > Misleading Error Message with Invalid Column and Group By > ---

[jira] [Resolved] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36867. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34244 [https://gith

[jira] [Resolved] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36914. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34236 [https://gith

[jira] [Assigned] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36914: --- Assignee: Huaxin Gao > Implement dropIndex and listIndexes in JDBC (MySQL dialect) > --

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {co

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {co

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {c

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {cod

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} \{{

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Environment: (was: {{Spark documentation mention this:}} {{https://spark.apache.org/docs/3.0.0/security.html}

[jira] [Created] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
zoli created SPARK-36988: Summary: What chipers spark support for internode communication? Key: SPARK-36988 URL: https://issues.apache.org/jira/browse/SPARK-36988 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string (was: ignoreCorruptFiles does w

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427672#comment-17427672 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: Apache Spark > Add Doc about FROM statement > > >

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: (was: Apache Spark) > Add Doc about FROM statement > --

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427673#comment-17427673 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull requ

  1   2   >