[jira] [Created] (SPARK-36984) Misleading Spark Streaming source documentation

2021-10-12 Thread Jira
Lukáš created SPARK-36984: - Summary: Misleading Spark Streaming source documentation Key: SPARK-36984 URL: https://issues.apache.org/jira/browse/SPARK-36984 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-36984) Misleading Spark Streaming source documentation

2021-10-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-36984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukáš updated SPARK-36984: -- Description: The documentation at [https://spark.apache.org/docs/latest/streaming-programming-guide.html#adva

[jira] [Updated] (SPARK-36984) Misleading Spark Streaming source documentation

2021-10-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-36984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukáš updated SPARK-36984: -- Attachment: docs_highlight.png > Misleading Spark Streaming source documentation > ---

[jira] [Commented] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

2021-10-12 Thread Kanishka Chauhan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427584#comment-17427584 ] Kanishka Chauhan commented on SPARK-24156: -- Hi [~tdas], We observed on Spark 2

[jira] [Assigned] (SPARK-36922) The SIGN/SIGNUM functions should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36922: Assignee: (was: Apache Spark) > The SIGN/SIGNUM functions should support ANSI interva

[jira] [Commented] (SPARK-36922) The SIGN/SIGNUM functions should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)

[jira] [Assigned] (SPARK-36922) The SIGN/SIGNUM functions should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36922: Assignee: Apache Spark > The SIGN/SIGNUM functions should support ANSI intervals > --

[jira] [Commented] (SPARK-36922) The SIGN/SIGNUM functions should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427594#comment-17427594 ] Apache Spark commented on SPARK-36922: -- User 'Peng-Lei' has created a pull request

[jira] [Created] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-36985: -- Summary: Future typing errors in pyspark.pandas Key: SPARK-36985 URL: https://issues.apache.org/jira/browse/SPARK-36985 Project: Spark Issue Type

[jira] [Commented] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Lauri Koobas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427599#comment-17427599 ] Lauri Koobas commented on SPARK-36971: -- How would I know the difference? It's a thi

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427600#comment-17427600 ] Maciej Szymkiewicz commented on SPARK-36985: cc [~hyukjin.kwon] [~XinrongM] 

[jira] [Commented] (SPARK-36921) The DIV function should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427611#comment-17427611 ] Apache Spark commented on SPARK-36921: -- User 'Peng-Lei' has created a pull request

[jira] [Commented] (SPARK-36921) The DIV function should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427610#comment-17427610 ] Apache Spark commented on SPARK-36921: -- User 'Peng-Lei' has created a pull request

[jira] [Assigned] (SPARK-36921) The DIV function should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36921: Assignee: Apache Spark > The DIV function should support ANSI intervals > ---

[jira] [Assigned] (SPARK-36921) The DIV function should support ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36921: Assignee: (was: Apache Spark) > The DIV function should support ANSI intervals >

[jira] [Comment Edited] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

2021-10-12 Thread Kanishka Chauhan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427584#comment-17427584 ] Kanishka Chauhan edited comment on SPARK-24156 at 10/12/21, 10:41 AM:

[jira] [Assigned] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36976: Assignee: Apache Spark > Add max_by/min_by API to SparkR > --

[jira] [Commented] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427616#comment-17427616 ] Apache Spark commented on SPARK-36976: -- User 'yoda-mon' has created a pull request

[jira] [Commented] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427618#comment-17427618 ] Apache Spark commented on SPARK-36976: -- User 'yoda-mon' has created a pull request

[jira] [Assigned] (SPARK-36976) Add max_by/min_by API to SparkR

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36976: Assignee: (was: Apache Spark) > Add max_by/min_by API to SparkR > ---

[jira] [Commented] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427624#comment-17427624 ] Apache Spark commented on SPARK-36949: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36949: Assignee: Apache Spark > Fix CREATE TABLE AS SELECT of ANSI intervals > -

[jira] [Assigned] (SPARK-36949) Fix CREATE TABLE AS SELECT of ANSI intervals

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36949: Assignee: (was: Apache Spark) > Fix CREATE TABLE AS SELECT of ANSI intervals > --

[jira] [Commented] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427627#comment-17427627 ] Apache Spark commented on SPARK-36979: -- User 'ulysses-you' has created a pull reque

[jira] [Created] (SPARK-36986) Improving external schema management flexibility

2021-10-12 Thread Rodrigo Boavida (Jira)
Rodrigo Boavida created SPARK-36986: --- Summary: Improving external schema management flexibility Key: SPARK-36986 URL: https://issues.apache.org/jira/browse/SPARK-36986 Project: Spark Issue

[jira] [Resolved] (SPARK-36972) Add max_by/min_by API to PySpark

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36972. Fix Version/s: 3.3.0 Assignee: Leona Yoda Resolution: Fixed Issue resolved

[jira] [Created] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread angerszhu (Jira)
angerszhu created SPARK-36987: - Summary: Add Doc about FROM statement Key: SPARK-36987 URL: https://issues.apache.org/jira/browse/SPARK-36987 Project: Spark Issue Type: Task Components:

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427673#comment-17427673 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427672#comment-17427672 ] Apache Spark commented on SPARK-36987: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: Apache Spark > Add Doc about FROM statement > > >

[jira] [Assigned] (SPARK-36987) Add Doc about FROM statement

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36987: Assignee: (was: Apache Spark) > Add Doc about FROM statement > --

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Summary: ignoreCorruptFiles does not work when schema change from int to string (was: ignoreCorruptFiles does w

[jira] [Created] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
zoli created SPARK-36988: Summary: What chipers spark support for internode communication? Key: SPARK-36988 URL: https://issues.apache.org/jira/browse/SPARK-36988 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Environment: (was: {{Spark documentation mention this:}} {{https://spark.apache.org/docs/3.0.0/security.html}

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} \{{

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {cod

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mention this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {c

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {co

[jira] [Updated] (SPARK-36988) What chipers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Description: {{Spark documentation mentions this:}} {{[https://spark.apache.org/docs/3.0.0/security.html]}} {co

[jira] [Assigned] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36914: --- Assignee: Huaxin Gao > Implement dropIndex and listIndexes in JDBC (MySQL dialect) > --

[jira] [Resolved] (SPARK-36914) Implement dropIndex and listIndexes in JDBC (MySQL dialect)

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36914. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34236 [https://gith

[jira] [Resolved] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36867. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34244 [https://gith

[jira] [Assigned] (SPARK-36867) Misleading Error Message with Invalid Column and Group By

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36867: --- Assignee: Wenchen Fan > Misleading Error Message with Invalid Column and Group By > ---

[jira] [Updated] (SPARK-36988) What ciphers spark support for internode communication?

2021-10-12 Thread zoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zoli updated SPARK-36988: - Summary: What ciphers spark support for internode communication? (was: What chipers spark support for internode

[jira] [Assigned] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-36970: Assignee: Yang Jie > Manual disabled format `B` for `date_format` function to compatibility with

[jira] [Resolved] (SPARK-36970) Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

2021-10-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36970. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34237 [https://github.com

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

2021-10-12 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427825#comment-17427825 ] Shardul Mahadik commented on SPARK-36877: - {quote} Getting RDD means the physica

[jira] [Updated] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Utkarsh Agarwal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Utkarsh Agarwal updated SPARK-36978: Description: [InferFiltersFromConstraints|https://github.com/apache/spark/blob/05c0fa57388

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427851#comment-17427851 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: Apache Spark > InferConstraints rule should create IsNotNull constraints on the

[jira] [Assigned] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36978: Assignee: (was: Apache Spark) > InferConstraints rule should create IsNotNull constra

[jira] [Commented] (SPARK-36978) InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427852#comment-17427852 ] Apache Spark commented on SPARK-36978: -- User 'utkarsh39' has created a pull request

[jira] [Created] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-36989: -- Summary: Migrate type hint data tests Key: SPARK-36989 URL: https://issues.apache.org/jira/browse/SPARK-36989 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427884#comment-17427884 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request f

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: Apache Spark > Allow Spark on Kube to operate without polling or watchers > ---

[jira] [Commented] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427886#comment-17427886 ] Apache Spark commented on SPARK-36462: -- User 'holdenk' has created a pull request f

[jira] [Assigned] (SPARK-36462) Allow Spark on Kube to operate without polling or watchers

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36462: Assignee: (was: Apache Spark) > Allow Spark on Kube to operate without polling or wat

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427888#comment-17427888 ] Maciej Szymkiewicz commented on SPARK-36989: Currently I am working on [some

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427889#comment-17427889 ] Maciej Szymkiewicz commented on SPARK-36989: FYI [~hyukjin.kwon] [~XinrongM]

[jira] [Comment Edited] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427889#comment-17427889 ] Maciej Szymkiewicz edited comment on SPARK-36989 at 10/12/21, 7:23 PM: ---

[jira] [Updated] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-36989: --- Description: Before the migration, {{pyspark-stubs}} contained a set of [data tests

[jira] [Created] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
Catalin Toda created SPARK-36990: Summary: Long columns cannot read columns with INT32 type in the parquet file Key: SPARK-36990 URL: https://issues.apache.org/jira/browse/SPARK-36990 Project: Spark

[jira] [Resolved] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

2021-10-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36951. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Environment: (was: Python repro: {code:java} import os from pyspark.sql.functions import * fr

[jira] [Updated] (SPARK-36990) Long columns cannot read columns with INT32 type in the parquet file

2021-10-12 Thread Catalin Toda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Catalin Toda updated SPARK-36990: - Description: The code below does not work on both Spark 3.1 and Spark 3.2. Part of the issue is

[jira] [Created] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36991: Summary: Inline type hints for spark/python/pyspark/sql/streaming.py Key: SPARK-36991 URL: https://issues.apache.org/jira/browse/SPARK-36991 Project: Spark

[jira] [Commented] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427922#comment-17427922 ] Xinrong Meng commented on SPARK-36991: -- I am working on this. > Inline type hints

[jira] [Resolved] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36979. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34260 [https://

[jira] [Assigned] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36979: - Assignee: XiDuo You > Add RewriteLateralSubquery rule into nonExcludableRules > ---

[jira] [Updated] (SPARK-36979) Add RewriteLateralSubquery rule into nonExcludableRules

2021-10-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36979: -- Issue Type: Bug (was: Improvement) > Add RewriteLateralSubquery rule into nonExcludableRules

[jira] [Commented] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427958#comment-17427958 ] Apache Spark commented on SPARK-23626: -- User 'JoshRosen' has created a pull request

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427967#comment-17427967 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: Apache Spark > Future typing errors in pyspark.pandas > ---

[jira] [Assigned] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36985: Assignee: (was: Apache Spark) > Future typing errors in pyspark.pandas >

[jira] [Commented] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427966#comment-17427966 ] Apache Spark commented on SPARK-36985: -- User 'ueshin' has created a pull request fo

[jira] [Resolved] (SPARK-36981) Upgrade joda-time to 2.10.12

2021-10-12 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36981. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in https://github.com/apache/

[jira] [Resolved] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36961. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34227 [https://gi

[jira] [Assigned] (SPARK-36961) Use PEP526 style variable type hints

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36961: Assignee: Takuya Ueshin > Use PEP526 style variable type hints >

[jira] [Commented] (SPARK-36989) Migrate type hint data tests

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427970#comment-17427970 ] Hyukjin Kwon commented on SPARK-36989: -- Adding mypy tests would be super awesome!

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Attachment: file2.parquet file1.parquet > ignoreCorruptFiles does not work when schema change fr

[jira] [Resolved] (SPARK-36985) Future typing errors in pyspark.pandas

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36985. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in https:

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Updated] (SPARK-36983) ignoreCorruptFiles does not work when schema change from int to string

2021-10-12 Thread mike (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mike updated SPARK-36983: - Description: Precondition: In folder A having two parquet files * File 1: have some columns and one of them is

[jira] [Commented] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427976#comment-17427976 ] Hyukjin Kwon commented on SPARK-36971: -- I suggest you do contact AWS or Databricks

[jira] [Resolved] (SPARK-36971) Query files directly with SQL is broken (with Glue)

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36971. -- Resolution: Invalid > Query files directly with SQL is broken (with Glue) > --

[jira] [Created] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36992: - Summary: Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray Key: SPARK-36992 URL: https://issues.apache.org/jira/browse/SPARK-36992 Projec

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: (was: Apache Spark) > Improve byte array sort perf by unify getPrefix funct

[jira] [Commented] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427981#comment-17427981 ] Apache Spark commented on SPARK-36992: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-36992) Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36992: Assignee: Apache Spark > Improve byte array sort perf by unify getPrefix function of UTF8

[jira] [Updated] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36900: - Fix Version/s: (was: 3.2.1) (was: 3.3.0) > "SPARK-36464: size returns

[jira] [Commented] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427992#comment-17427992 ] Hyukjin Kwon commented on SPARK-36900: -- Reverted in: https://github.com/apache/spa

[jira] [Reopened] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-36900: -- Assignee: (was: Sean R. Owen) > "SPARK-36464: size returns correct positive number even

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: (was: Apache Spark) > "SPARK-36464: size returns correct positive number ev

[jira] [Resolved] (SPARK-36954) Fast fail with explicit err msg when calling withWatermark on non-streaming dataset

2021-10-12 Thread huangtengfei (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtengfei resolved SPARK-36954. -- Resolution: Not A Problem > Fast fail with explicit err msg when calling withWatermark on non-

[jira] [Assigned] (SPARK-36900) "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

2021-10-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36900: Assignee: Apache Spark > "SPARK-36464: size returns correct positive number even with ove

[jira] [Resolved] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36794. - Resolution: Fixed Issue resolved by pull request 34247 [https://github.com/apache/spark/pull/342

[jira] [Updated] (SPARK-36794) Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join

2021-10-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-36794: Summary: Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join (was:

[jira] [Resolved] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36953. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34219 [https://gi

[jira] [Assigned] (SPARK-36953) Expose SQL state and error class in PySpark exceptions

2021-10-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36953: Assignee: Hyukjin Kwon > Expose SQL state and error class in PySpark exceptions > ---

[jira] [Created] (SPARK-36993) Fix json_tupe throw NPE if fields exist no foldable null column

2021-10-12 Thread XiDuo You (Jira)
XiDuo You created SPARK-36993: - Summary: Fix json_tupe throw NPE if fields exist no foldable null column Key: SPARK-36993 URL: https://issues.apache.org/jira/browse/SPARK-36993 Project: Spark Is

  1   2   >