[jira] [Commented] (SPARK-36387) Fix Series.astype from datetime to nullable string

2021-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397828#comment-17397828 ] Hyukjin Kwon commented on SPARK-36387: -- If he's inactive, let's go ahead. [~itholic]. We should

[jira] [Commented] (SPARK-36387) Fix Series.astype from datetime to nullable string

2021-08-11 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397825#comment-17397825 ] Haejoon Lee commented on SPARK-36387: - [~dc-heros], are you still working on this?? No need to

[jira] [Updated] (SPARK-36387) Fix Series.astype from datetime to nullable string

2021-08-11 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-36387: Attachment: image-2021-08-12-14-24-31-321.png > Fix Series.astype from datetime to nullable

[jira] [Resolved] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36479. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33707

[jira] [Assigned] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36479: --- Assignee: Wenchen Fan > improve datetime test coverage in SQL files >

[jira] [Updated] (SPARK-33807) Data Source V2: Remove read specific distributions

2021-08-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-33807: Affects Version/s: (was: 3.2.0) 3.3.0 > Data Source V2: Remove read

[jira] [Assigned] (SPARK-36486) Support type constructed string as dat time interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36486: Assignee: (was: Apache Spark) > Support type constructed string as dat time interval

[jira] [Assigned] (SPARK-36486) Support type constructed string as dat time interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36486: Assignee: Apache Spark > Support type constructed string as dat time interval >

[jira] [Commented] (SPARK-36486) Support type constructed string as dat time interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397804#comment-17397804 ] Apache Spark commented on SPARK-36486: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36485) Support cast type constructed string to year month interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1739#comment-1739 ] Apache Spark commented on SPARK-36485: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36485) Support cast type constructed string to year month interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36485: Assignee: (was: Apache Spark) > Support cast type constructed string to year month

[jira] [Assigned] (SPARK-36485) Support cast type constructed string to year month interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36485: Assignee: Apache Spark > Support cast type constructed string to year month interval >

[jira] [Commented] (SPARK-36485) Support cast type constructed string to year month interval

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397778#comment-17397778 ] Apache Spark commented on SPARK-36485: -- User 'AngersZh' has created a pull request for this

[jira] [Created] (SPARK-36486) Support type constructed string as dat time interval

2021-08-11 Thread angerszhu (Jira)
angerszhu created SPARK-36486: - Summary: Support type constructed string as dat time interval Key: SPARK-36486 URL: https://issues.apache.org/jira/browse/SPARK-36486 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36480: --- Assignee: Jungtaek Lim > SessionWindowStateStoreSaveExec should not filter input rows

[jira] [Commented] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397775#comment-17397775 ] L. C. Hsieh commented on SPARK-36480: - The issue was resolved at

[jira] [Resolved] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36480. - Resolution: Fixed > SessionWindowStateStoreSaveExec should not filter input rows against

[jira] [Updated] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-36480: Fix Version/s: 3.2.0 > SessionWindowStateStoreSaveExec should not filter input rows against

[jira] [Created] (SPARK-36485) Support cast type constructed string to year month interval

2021-08-11 Thread angerszhu (Jira)
angerszhu created SPARK-36485: - Summary: Support cast type constructed string to year month interval Key: SPARK-36485 URL: https://issues.apache.org/jira/browse/SPARK-36485 Project: Spark Issue

[jira] [Assigned] (SPARK-32986) Add bucket scan info in explain output of FileSourceScanExec

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32986: - Assignee: Cheng Su > Add bucket scan info in explain output of FileSourceScanExec >

[jira] [Resolved] (SPARK-32986) Add bucket scan info in explain output of FileSourceScanExec

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32986. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33698

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397747#comment-17397747 ] Dongjoon Hyun commented on SPARK-18105: --- I did only the above code I posted because you wrote like

[jira] [Updated] (SPARK-34081) Only pushdown LeftSemi/LeftAnti over Aggregate if join can be planned as broadcast join

2021-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-34081: --- Issue Type: Improvement (was: Bug) > Only pushdown LeftSemi/LeftAnti over Aggregate if

[jira] [Commented] (SPARK-32285) Add PySpark support for nested timestamps with arrow

2021-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397729#comment-17397729 ] Hyukjin Kwon commented on SPARK-32285: -- Arrow version is 2.0.0 now. Can we fix this ticket? > Add

[jira] [Commented] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

2021-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397725#comment-17397725 ] Mridul Muralidharan commented on SPARK-36378: - Thanks for marking this resolved [~Ngone51] !

[jira] [Created] (SPARK-36484) Handle Stale block fetch failure on the client side by not retrying the requests

2021-08-11 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36484: --- Summary: Handle Stale block fetch failure on the client side by not retrying the requests Key: SPARK-36484 URL:

[jira] [Resolved] (SPARK-36481) Expose LogisticRegression.setInitialModel

2021-08-11 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-36481. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33710

[jira] [Assigned] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36483: Assignee: Apache Spark > Fix intermittent test failure due to netty dependency version

[jira] [Assigned] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36483: Assignee: (was: Apache Spark) > Fix intermittent test failure due to netty

[jira] [Commented] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397639#comment-17397639 ] Apache Spark commented on SPARK-36483: -- User 'Victsm' has created a pull request for this issue:

[jira] [Updated] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

2021-08-11 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-36483: - Description: In SPARK-35132, Spark's netty dependency version was bumped from 4.1.51 to 4.1.63. Since

[jira] [Created] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

2021-08-11 Thread Min Shen (Jira)
Min Shen created SPARK-36483: Summary: Fix intermittent test failure due to netty dependency version bump Key: SPARK-36483 URL: https://issues.apache.org/jira/browse/SPARK-36483 Project: Spark

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-08-11 Thread Cameron Todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397603#comment-17397603 ] Cameron Todd commented on SPARK-18105: -- Good to hear. Also the count of 136,935,074 is right. 

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397585#comment-17397585 ] Dongjoon Hyun commented on SPARK-18105: --- BTW, I checked that a single zip file contains 157 snappy

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397584#comment-17397584 ] Dongjoon Hyun commented on SPARK-18105: --- Thank you, [~cameron.todd]. I downloaded and ran the test

[jira] [Updated] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34827: -- Affects Version/s: 3.3.0 > Support fetching shuffle blocks in batch with i/o encryption >

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397569#comment-17397569 ] Dongjoon Hyun commented on SPARK-34827: --- Thank you. I moved the Target Version to 3.3.0. >

[jira] [Updated] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34827: -- Target Version/s: 3.3.0 (was: 3.2.0) > Support fetching shuffle blocks in batch with i/o

[jira] [Commented] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397563#comment-17397563 ] Apache Spark commented on SPARK-36378: -- User 'Victsm' has created a pull request for this issue:

[jira] [Resolved] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36482. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33712

[jira] [Assigned] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36482: - Assignee: William Hyun > Bump orc to 1.6.10 > -- > >

[jira] [Assigned] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36482: Assignee: Apache Spark > Bump orc to 1.6.10 > -- > >

[jira] [Commented] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397486#comment-17397486 ] Apache Spark commented on SPARK-36482: -- User 'williamhyun' has created a pull request for this

[jira] [Assigned] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36482: Assignee: (was: Apache Spark) > Bump orc to 1.6.10 > -- > >

[jira] [Created] (SPARK-36482) Bump orc to 1.6.10

2021-08-11 Thread William Hyun (Jira)
William Hyun created SPARK-36482: Summary: Bump orc to 1.6.10 Key: SPARK-36482 URL: https://issues.apache.org/jira/browse/SPARK-36482 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

2021-08-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397461#comment-17397461 ] Thomas Graves commented on SPARK-36446: --- [~adamkennedy77] ^ > YARN shuffle server restart crashes

[jira] [Assigned] (SPARK-36481) Expose LogisticRegression.setInitialModel

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36481: Assignee: Sean R. Owen (was: Apache Spark) > Expose LogisticRegression.setInitialModel

[jira] [Assigned] (SPARK-36481) Expose LogisticRegression.setInitialModel

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36481: Assignee: Apache Spark (was: Sean R. Owen) > Expose LogisticRegression.setInitialModel

[jira] [Commented] (SPARK-36481) Expose LogisticRegression.setInitialModel

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397445#comment-17397445 ] Apache Spark commented on SPARK-36481: -- User 'srowen' has created a pull request for this issue:

[jira] [Created] (SPARK-36481) Expose LogisticRegression.setInitialModel

2021-08-11 Thread Sean R. Owen (Jira)
Sean R. Owen created SPARK-36481: Summary: Expose LogisticRegression.setInitialModel Key: SPARK-36481 URL: https://issues.apache.org/jira/browse/SPARK-36481 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36360) Delete appName from StreamingSource

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. However, the

[jira] [Updated] (SPARK-36360) Delete appName from StreamingSource

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. The appName should

[jira] [Updated] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-36478: Description: Removes outer join if all grouping and aggregate expressions are from the streamed side.

[jira] [Updated] (SPARK-36360) Delete appName from StreamingSource

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. This is not

[jira] [Updated] (SPARK-36360) Delete appName from StreamingSource

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Summary: Delete appName from StreamingSource (was: StreamingSource duplicates appName) >

[jira] [Updated] (SPARK-36360) StreamingSource duplicates appName

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. This is not

[jira] [Updated] (SPARK-36360) StreamingSource duplicates appName

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Description: The StreamingSource includes the appName in its sourceName. This is not

[jira] [Updated] (SPARK-36360) StreamingSource duplicates appName

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Priority: Blocker (was: Trivial) > StreamingSource duplicates appName >

[jira] [Updated] (SPARK-36360) StreamingSource duplicates appName

2021-08-11 Thread Marcel Neumann (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Neumann updated SPARK-36360: --- Priority: Trivial (was: Minor) > StreamingSource duplicates appName >

[jira] [Updated] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-36478: Description: Removes outer join if all grouping and aggregate expressions are from the streamed side.

[jira] [Assigned] (SPARK-35030) ANSI SQL compliance

2021-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-35030: -- Assignee: Apache Spark > ANSI SQL compliance > --- > >

[jira] [Assigned] (SPARK-36418) Use CAST in parsing of dates/timestamps with default pattern

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36418: Assignee: (was: Apache Spark) > Use CAST in parsing of dates/timestamps with default

[jira] [Assigned] (SPARK-36418) Use CAST in parsing of dates/timestamps with default pattern

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36418: Assignee: Apache Spark > Use CAST in parsing of dates/timestamps with default pattern >

[jira] [Commented] (SPARK-36418) Use CAST in parsing of dates/timestamps with default pattern

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397376#comment-17397376 ] Apache Spark commented on SPARK-36418: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36474) Mention pandas API on Spark in Spark overview pages

2021-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36474: Assignee: Hyukjin Kwon > Mention pandas API on Spark in Spark overview pages >

[jira] [Resolved] (SPARK-36474) Mention pandas API on Spark in Spark overview pages

2021-08-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36474. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33699

[jira] [Resolved] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

2021-08-11 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi resolved SPARK-36378. -- Fix Version/s: 3.3.0 3.2.0 Assignee: Min Shen Resolution: Fixed Issue

[jira] [Commented] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397336#comment-17397336 ] Apache Spark commented on SPARK-36480: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Assigned] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36480: Assignee: Apache Spark > SessionWindowStateStoreSaveExec should not filter input rows

[jira] [Assigned] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36480: Assignee: (was: Apache Spark) > SessionWindowStateStoreSaveExec should not filter

[jira] [Commented] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397334#comment-17397334 ] Apache Spark commented on SPARK-36480: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Created] (SPARK-36480) SessionWindowStateStoreSaveExec should not filter input rows against watermark

2021-08-11 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-36480: Summary: SessionWindowStateStoreSaveExec should not filter input rows against watermark Key: SPARK-36480 URL: https://issues.apache.org/jira/browse/SPARK-36480

[jira] [Resolved] (SPARK-36132) Support initial state for flatMapGroupsWithState in batch mode

2021-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36132. Resolution: Done This issue is resolved in https://github.com/apache/spark/pull/6 >

[jira] [Assigned] (SPARK-36132) Support initial state for flatMapGroupsWithState in batch mode

2021-08-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36132: -- Assignee: Rahul Shivu Mahadev > Support initial state for flatMapGroupsWithState in

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2021-08-11 Thread Cameron Todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397301#comment-17397301 ] Cameron Todd commented on SPARK-18105: -- Ok I added the zip file on this public S3 bucket, it holds

[jira] [Updated] (SPARK-36413) Spark and Hive are inconsistent in inferring the expression type in the view, resulting in AnalysisException

2021-08-11 Thread bingfeng.guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bingfeng.guo updated SPARK-36413: - Priority: Blocker (was: Critical) > Spark and Hive are inconsistent in inferring the

[jira] [Commented] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397279#comment-17397279 ] Apache Spark commented on SPARK-36478: -- User 'wankunde' has created a pull request for this issue:

[jira] [Commented] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397278#comment-17397278 ] Apache Spark commented on SPARK-36479: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36479: Assignee: Apache Spark > improve datetime test coverage in SQL files >

[jira] [Commented] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397276#comment-17397276 ] Apache Spark commented on SPARK-36478: -- User 'wankunde' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36478: Assignee: Apache Spark > Removes outer join if all grouping and aggregate expressions

[jira] [Assigned] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36479: Assignee: (was: Apache Spark) > improve datetime test coverage in SQL files >

[jira] [Assigned] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36478: Assignee: (was: Apache Spark) > Removes outer join if all grouping and aggregate

[jira] [Created] (SPARK-36479) improve datetime test coverage in SQL files

2021-08-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-36479: --- Summary: improve datetime test coverage in SQL files Key: SPARK-36479 URL: https://issues.apache.org/jira/browse/SPARK-36479 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-36478) Removes outer join if all grouping and aggregate expressions are from the streamed side

2021-08-11 Thread Wan Kun (Jira)
Wan Kun created SPARK-36478: --- Summary: Removes outer join if all grouping and aggregate expressions are from the streamed side Key: SPARK-36478 URL: https://issues.apache.org/jira/browse/SPARK-36478

[jira] [Resolved] (SPARK-36468) Update docs about ANSI interval literals

2021-08-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36468. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33693

[jira] [Updated] (SPARK-36472) Improve SQL syntax for MERGE

2021-08-11 Thread Denis Krivenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Krivenko updated SPARK-36472: --- Description: Existing SQL syntax for *MEGRE* (see Delta Lake examples

[jira] [Commented] (SPARK-36353) RemoveNoopOperators should keep output schema

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397236#comment-17397236 ] Apache Spark commented on SPARK-36353: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36353) RemoveNoopOperators should keep output schema

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397238#comment-17397238 ] Apache Spark commented on SPARK-36353: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36477) Infering schema from JSON file shall respect ignoreCorruptFiles and handle IOE

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36477: Assignee: Apache Spark > Infering schema from JSON file shall respect ignoreCorruptFiles

[jira] [Commented] (SPARK-36477) Infering schema from JSON file shall respect ignoreCorruptFiles and handle IOE

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397237#comment-17397237 ] Apache Spark commented on SPARK-36477: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36477) Infering schema from JSON file shall respect ignoreCorruptFiles and handle IOE

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36477: Assignee: (was: Apache Spark) > Infering schema from JSON file shall respect

[jira] [Created] (SPARK-36477) Infering schema from JSON file shall respect ignoreCorruptFiles and handle IOE

2021-08-11 Thread Kent Yao (Jira)
Kent Yao created SPARK-36477: Summary: Infering schema from JSON file shall respect ignoreCorruptFiles and handle IOE Key: SPARK-36477 URL: https://issues.apache.org/jira/browse/SPARK-36477 Project:

[jira] [Commented] (SPARK-36352) Spark should check result plan's output schema name

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397206#comment-17397206 ] Apache Spark commented on SPARK-36352: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36352) Spark should check result plan's output schema name

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397205#comment-17397205 ] Apache Spark commented on SPARK-36352: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36475) Add doc about spark.shuffle.service.fetch.rdd.enabled

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397152#comment-17397152 ] Apache Spark commented on SPARK-36475: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36475) Add doc about spark.shuffle.service.fetch.rdd.enabled

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397151#comment-17397151 ] Apache Spark commented on SPARK-36475: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36475) Add doc about spark.shuffle.service.fetch.rdd.enabled

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36475: Assignee: (was: Apache Spark) > Add doc about

[jira] [Assigned] (SPARK-36475) Add doc about spark.shuffle.service.fetch.rdd.enabled

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36475: Assignee: Apache Spark > Add doc about spark.shuffle.service.fetch.rdd.enabled >

[jira] [Commented] (SPARK-34808) Removes outer join if it only has distinct on streamed side

2021-08-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397148#comment-17397148 ] Apache Spark commented on SPARK-34808: -- User 'wankunde' has created a pull request for this issue:

  1   2   >