[jira] [Created] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

Yuming Wang created SPARK-38771: --- Summary: Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin Key: SPARK-38771 URL: https://issues.apache.org/jira/browse/SPARK-38771

[jira] [Updated] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38771: Attachment: Special case.png > Introduce adaptive Bloom filter Join to reduce spilling to disk whe

[jira] [Updated] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38771: Description: Insert a bloom filter on one side of the join if it may be spill when sorting and th

[jira] [Created] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

Yuming Wang created SPARK-38772: --- Summary: Formatting the log plan in AdaptiveSparkPlanExec Key: SPARK-38772 URL: https://issues.apache.org/jira/browse/SPARK-38772 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516246#comment-17516246 ] Apache Spark commented on SPARK-38772: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38772: Assignee: (was: Apache Spark) > Formatting the log plan in AdaptiveSparkPlanExec > --

[jira] [Assigned] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38772: Assignee: Apache Spark > Formatting the log plan in AdaptiveSparkPlanExec > -

[jira] [Commented] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516247#comment-17516247 ] Apache Spark commented on SPARK-38772: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-38763) Pandas API on spark Can`t apply lamda to columns.

[ https://issues.apache.org/jira/browse/SPARK-38763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516259#comment-17516259 ] Bjørn Jørgensen commented on SPARK-38763: - [~XinrongM] I did replace frame.py wi

[jira] [Created] (SPARK-38773) Correct the Union output partitioning and ordering

XiDuo You created SPARK-38773: - Summary: Correct the Union output partitioning and ordering Key: SPARK-38773 URL: https://issues.apache.org/jira/browse/SPARK-38773 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516274#comment-17516274 ] Apache Spark commented on SPARK-38771: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38771: Assignee: Apache Spark > Introduce adaptive Bloom filter Join to reduce spilling to disk

[jira] [Assigned] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38771: Assignee: (was: Apache Spark) > Introduce adaptive Bloom filter Join to reduce spilli

[jira] [Assigned] (SPARK-38771) Introduce adaptive Bloom filter Join to reduce spilling to disk when doing SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-38771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38771: Assignee: Apache Spark > Introduce adaptive Bloom filter Join to reduce spilling to disk

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516280#comment-17516280 ] Cheng Pan commented on SPARK-30602: --- Hi [~mshen] [~mridulm80] Why your reporter shows

[jira] [Comment Edited] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516280#comment-17516280 ] Cheng Pan edited comment on SPARK-30602 at 4/2/22 12:10 PM:

[jira] [Created] (SPARK-38774) impl Series.autocorr

zhengruifeng created SPARK-38774: Summary: impl Series.autocorr Key: SPARK-38774 URL: https://issues.apache.org/jira/browse/SPARK-38774 Project: Spark Issue Type: Sub-task Component

[jira] [Commented] (SPARK-38774) impl Series.autocorr

[ https://issues.apache.org/jira/browse/SPARK-38774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516292#comment-17516292 ] Apache Spark commented on SPARK-38774: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-38774) impl Series.autocorr

[ https://issues.apache.org/jira/browse/SPARK-38774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38774: Assignee: (was: Apache Spark) > impl Series.autocorr > > >

[jira] [Assigned] (SPARK-38774) impl Series.autocorr

[ https://issues.apache.org/jira/browse/SPARK-38774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38774: Assignee: Apache Spark > impl Series.autocorr > > >

[jira] [Commented] (SPARK-32268) Bloom Filter Join

[ https://issues.apache.org/jira/browse/SPARK-32268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516293#comment-17516293 ] Apache Spark commented on SPARK-32268: -- User 'Flyangz' has created a pull request f

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516298#comment-17516298 ] Min Shen commented on SPARK-30602: -- [~pan3793] , I guess you were referring to the earl

[jira] [Comment Edited] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516298#comment-17516298 ] Min Shen edited comment on SPARK-30602 at 4/2/22 1:02 PM: -- [~pa

[jira] [Resolved] (SPARK-38757) Update the Oracle docker image version used for test and integration

[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38757. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36036 [https://gith

[jira] [Assigned] (SPARK-38757) Update the Oracle docker image version used for test and integration

[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38757: --- Assignee: Luca Canali > Update the Oracle docker image version used for test and integratio

[jira] [Assigned] (SPARK-38669) Validate input dataset of ml.clustering

[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38669: Assignee: zhengruifeng > Validate input dataset of ml.clustering > --

[jira] [Resolved] (SPARK-38669) Validate input dataset of ml.clustering

[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38669. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35983 [https://gi

[jira] [Updated] (SPARK-38669) Validate input dataset of ml.clustering

[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-38669: - Priority: Minor (was: Major) > Validate input dataset of ml.clustering > --

[jira] [Updated] (SPARK-38584) Unify the data validation

[ https://issues.apache.org/jira/browse/SPARK-38584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-38584: - Priority: Minor (was: Major) > Unify the data validation > - > >

[jira] [Resolved] (SPARK-38584) Unify the data validation

[ https://issues.apache.org/jira/browse/SPARK-38584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38584. -- Fix Version/s: 3.4.0 Resolution: Fixed > Unify the data validation > --

[jira] [Updated] (SPARK-38643) Validate input dataset of ml.regression

[ https://issues.apache.org/jira/browse/SPARK-38643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-38643: - Priority: Minor (was: Major) > Validate input dataset of ml.regression > --

[jira] [Updated] (SPARK-38588) Validate input dataset of ml.classification

[ https://issues.apache.org/jira/browse/SPARK-38588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-38588: - Priority: Minor (was: Major) > Validate input dataset of ml.classification > --

[jira] [Assigned] (SPARK-38756) Clean up useless security configs in `TransportConf`

[ https://issues.apache.org/jira/browse/SPARK-38756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38756: Assignee: Yang Jie > Clean up useless security configs in `TransportConf` > -

[jira] [Resolved] (SPARK-38756) Clean up useless security configs in `TransportConf`

[ https://issues.apache.org/jira/browse/SPARK-38756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38756. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36035 [https://gi

[jira] [Resolved] (SPARK-38661) [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests

[ https://issues.apache.org/jira/browse/SPARK-38661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38661. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35976 [https://gi

[jira] [Updated] (SPARK-38675) Race condition in BlockInfoManager during unlock

[ https://issues.apache.org/jira/browse/SPARK-38675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38675: -- Priority: Blocker (was: Major) > Race condition in BlockInfoManager during unlock > -

[jira] [Updated] (SPARK-38675) Race condition in BlockInfoManager during unlock

[ https://issues.apache.org/jira/browse/SPARK-38675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38675: -- Target Version/s: 3.3.0 > Race condition in BlockInfoManager during unlock > -

[jira] [Updated] (SPARK-38757) Update the Oracle docker image version used for test and integration

[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38757: -- Component/s: Tests (was: Build) > Update the Oracle docker image version

[jira] [Assigned] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38686: Assignee: Xinrong Meng > Implement `keep` parameter of `(Index/MultiIndex).drop_duplicate

[jira] [Resolved] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38686. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36006 [https://gi

[jira] [Resolved] (SPARK-38479) Add `Series.duplicated` to indicate duplicate Series values.

[ https://issues.apache.org/jira/browse/SPARK-38479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38479. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35782 [https://gi

[jira] [Assigned] (SPARK-38479) Add `Series.duplicated` to indicate duplicate Series values.

[ https://issues.apache.org/jira/browse/SPARK-38479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38479: Assignee: Xinrong Meng > Add `Series.duplicated` to indicate duplicate Series values. > -

[jira] [Resolved] (SPARK-38763) Pandas API on spark Can`t apply lamda to columns.

[ https://issues.apache.org/jira/browse/SPARK-38763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38763. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36042 [https://gi

[jira] [Assigned] (SPARK-38763) Pandas API on spark Can`t apply lamda to columns.

[ https://issues.apache.org/jira/browse/SPARK-38763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38763: Assignee: Xinrong Meng > Pandas API on spark Can`t apply lamda to columns. > --

[jira] [Commented] (SPARK-38769) [SQL] behavior of schema_of_json not same with 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-38769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516416#comment-17516416 ] Hyukjin Kwon commented on SPARK-38769: -- This is meant to work together with {{from_

[jira] [Updated] (SPARK-38764) spark thrift server issue: Length field is empty for varchar fields

[ https://issues.apache.org/jira/browse/SPARK-38764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38764: - Priority: Major (was: Critical) > spark thrift server issue: Length field is empty for varchar

[jira] (SPARK-38769) [SQL] behavior of schema_of_json not same with 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-38769 ] gabrywu deleted comment on SPARK-38769: - was (Author: gabry.wu): [~maxgekk]  > [SQL] behavior of schema_of_json not same with 2.4.0 > > >

[jira] [Comment Edited] (SPARK-38769) [SQL] behavior of schema_of_json not same with 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-38769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516421#comment-17516421 ] gabrywu edited comment on SPARK-38769 at 4/3/22 2:20 AM: - [~hyuk

[jira] [Commented] (SPARK-38769) [SQL] behavior of schema_of_json not same with 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-38769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516421#comment-17516421 ] gabrywu commented on SPARK-38769: - nomatter which UDF to work together, I believe we sho

[jira] [Assigned] (SPARK-38770) Simply steps to re write primary resource in k8s spark application

[ https://issues.apache.org/jira/browse/SPARK-38770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38770: - Assignee: qian > Simply steps to re write primary resource in k8s spark application > -

[jira] [Resolved] (SPARK-38770) Simply steps to re write primary resource in k8s spark application

[ https://issues.apache.org/jira/browse/SPARK-38770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38770. --- Resolution: Fixed Issue resolved by pull request 36044 [https://github.com/apache/spark/pull

[jira] [Updated] (SPARK-38770) Remove renameMainAppResource from baseDriverContainer

[ https://issues.apache.org/jira/browse/SPARK-38770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38770: -- Summary: Remove renameMainAppResource from baseDriverContainer (was: Simply steps to re write

[jira] [Updated] (SPARK-38770) Remove renameMainAppResource from baseDriverContainer

[ https://issues.apache.org/jira/browse/SPARK-38770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38770: -- Affects Version/s: 3.4.0 (was: 3.1.0) (wa

[jira] [Updated] (SPARK-38770) Remove renameMainAppResource from baseDriverContainer

[ https://issues.apache.org/jira/browse/SPARK-38770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38770: -- Priority: Minor (was: Major) > Remove renameMainAppResource from baseDriverContainer > --

[jira] [Created] (SPARK-38775) cleanup validation functions

zhengruifeng created SPARK-38775: Summary: cleanup validation functions Key: SPARK-38775 URL: https://issues.apache.org/jira/browse/SPARK-38775 Project: Spark Issue Type: Sub-task C

[jira] [Assigned] (SPARK-38775) cleanup validation functions

[ https://issues.apache.org/jira/browse/SPARK-38775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-38775: Assignee: zhengruifeng > cleanup validation functions > > >

[jira] [Resolved] (SPARK-38594) Change to use `NettyUtils` to create `EventLoop` and `ChannelClass` in RBackend

[ https://issues.apache.org/jira/browse/SPARK-38594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38594. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35910 [https://

[jira] [Assigned] (SPARK-38594) Change to use `NettyUtils` to create `EventLoop` and `ChannelClass` in RBackend

[ https://issues.apache.org/jira/browse/SPARK-38594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38594: - Assignee: Yang Jie > Change to use `NettyUtils` to create `EventLoop` and `ChannelClass

[jira] [Assigned] (SPARK-38775) cleanup validation functions

[ https://issues.apache.org/jira/browse/SPARK-38775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38775: Assignee: Apache Spark (was: zhengruifeng) > cleanup validation functions >

[jira] [Assigned] (SPARK-38775) cleanup validation functions

[ https://issues.apache.org/jira/browse/SPARK-38775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38775: Assignee: zhengruifeng (was: Apache Spark) > cleanup validation functions >

[jira] [Commented] (SPARK-38775) cleanup validation functions

[ https://issues.apache.org/jira/browse/SPARK-38775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516435#comment-17516435 ] Apache Spark commented on SPARK-38775: -- User 'zhengruifeng' has created a pull requ

[jira] [Updated] (SPARK-38446) Deadlock between ExecutorClassLoader and FileDownloadCallback caused by Log4j

[ https://issues.apache.org/jira/browse/SPARK-38446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38446: -- Issue Type: Bug (was: Suitable Name Search) > Deadlock between ExecutorClassLoader and FileDo

[jira] [Created] (SPARK-38776) Flaky test: ALSSuite.'ALS validate input dataset'

Dongjoon Hyun created SPARK-38776: - Summary: Flaky test: ALSSuite.'ALS validate input dataset' Key: SPARK-38776 URL: https://issues.apache.org/jira/browse/SPARK-38776 Project: Spark Issue Typ