[jira] [Updated] (SPARK-31848) DAGSchedulerSuite: Break down the very huge test files, each test suite should focus on one or several major features, but not all the related behaviors

2020-09-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31848: --- Summary: DAGSchedulerSuite: Break down the very huge test files, each test suite should focus on one

[jira] [Assigned] (SPARK-31848) DAGSchedulerSuite: Break down the very huge test files, each test suite should focus on one or several major features, but not all the related behaviors

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31848: Assignee: Apache Spark > DAGSchedulerSuite: Break down the very huge test files, each tes

[jira] [Commented] (SPARK-31848) DAGSchedulerSuite: Break down the very huge test files, each test suite should focus on one or several major features, but not all the related behaviors

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195242#comment-17195242 ] Apache Spark commented on SPARK-31848: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-31848) DAGSchedulerSuite: Break down the very huge test files, each test suite should focus on one or several major features, but not all the related behaviors

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31848: Assignee: (was: Apache Spark) > DAGSchedulerSuite: Break down the very huge test file

[jira] [Updated] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-32872: --- Affects Version/s: 1.4.1 1.5.2 1.6.3 > BytesToBytesMap

[jira] [Commented] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195252#comment-17195252 ] Ankur Dave commented on SPARK-32872: Thanks, [~dongjoon]! Based on a quick look at t

[jira] [Comment Edited] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195252#comment-17195252 ] Ankur Dave edited comment on SPARK-32872 at 9/14/20, 7:30 AM:

[jira] [Comment Edited] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195252#comment-17195252 ] Ankur Dave edited comment on SPARK-32872 at 9/14/20, 7:37 AM:

[jira] [Updated] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Ankur Dave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-32872: --- Affects Version/s: (was: 1.5.2) (was: 1.4.1) > BytesToBytesMap at MAX

[jira] [Assigned] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-09-14 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-32708: -- Assignee: Mingjia Liu > Query optimization fails to reuse exchange with DataSourceV2

[jira] [Resolved] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-09-14 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-32708. Resolution: Fixed The issue is resolved in https://github.com/apache/spark/pull/29564 > Q

[jira] [Reopened] (SPARK-32542) Add an optimizer rule to split an Expand into multiple Expands for aggregates

2020-09-14 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro reopened SPARK-32542: -- > Add an optimizer rule to split an Expand into multiple Expands for aggregates >

[jira] [Resolved] (SPARK-32542) Add an optimizer rule to split an Expand into multiple Expands for aggregates

2020-09-14 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32542. -- Resolution: Won't Fix > Add an optimizer rule to split an Expand into multiple Expands

[jira] [Updated] (SPARK-32875) TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method.

2020-09-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-32875: --- Summary: TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract t

[jira] [Created] (SPARK-32875) TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the

2020-09-14 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-32875: -- Summary: TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the Key: SPARK-32875 URL: https://issues.apache.org/jira/browse/SPARK-32875

[jira] [Commented] (SPARK-32790) FINISHED state of application is not final

2020-09-14 Thread Rosie Bloxsom (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195288#comment-17195288 ] Rosie Bloxsom commented on SPARK-32790: --- Thanks [~dongjoon], Re. docs: * +1 on t

[jira] [Updated] (SPARK-32790) FINISHED state of application is not final

2020-09-14 Thread Rosie Bloxsom (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rosie Bloxsom updated SPARK-32790: -- Description: If you launch an application with SparkLauncher.startApplication, and pass a lis

[jira] [Resolved] (SPARK-32854) Avoid per-row join type check in stream-stream join

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32854. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29724 [https://gith

[jira] [Assigned] (SPARK-32854) Avoid per-row join type check in stream-stream join

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32854: --- Assignee: Cheng Su > Avoid per-row join type check in stream-stream join >

[jira] [Assigned] (SPARK-32844) Make `DataFrameReader.table` take the specified options for datasource v1

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32844: --- Assignee: Yuanjian Li > Make `DataFrameReader.table` take the specified options for datasou

[jira] [Resolved] (SPARK-32844) Make `DataFrameReader.table` take the specified options for datasource v1

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32844. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29712 [https://gith

[jira] [Updated] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32876: - Priority: Minor (was: Major) > Change default fallback versions in HiveExternalCatalogVersionsS

[jira] [Updated] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32876: - Issue Type: Test (was: Bug) > Change default fallback versions in HiveExternalCatalogVersionsSu

[jira] [Created] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32876: Summary: Change default fallback versions in HiveExternalCatalogVersionsSuite Key: SPARK-32876 URL: https://issues.apache.org/jira/browse/SPARK-32876 Project: Spark

[jira] [Created] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

2020-09-14 Thread ulysses you (Jira)
ulysses you created SPARK-32877: --- Summary: Fix Hive UDF not support decimal type in complex type Key: SPARK-32877 URL: https://issues.apache.org/jira/browse/SPARK-32877 Project: Spark Issue Typ

[jira] [Updated] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32876: - Affects Version/s: 3.1.0 > Change default fallback versions in HiveExternalCatalogVersionsSuite

[jira] [Assigned] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32876: Assignee: Apache Spark > Change default fallback versions in HiveExternalCatalogVersionsS

[jira] [Commented] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195363#comment-17195363 ] Apache Spark commented on SPARK-32876: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32876: Assignee: (was: Apache Spark) > Change default fallback versions in HiveExternalCatal

[jira] [Commented] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195364#comment-17195364 ] Apache Spark commented on SPARK-32877: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32877: Assignee: (was: Apache Spark) > Fix Hive UDF not support decimal type in complex type

[jira] [Assigned] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32877: Assignee: Apache Spark > Fix Hive UDF not support decimal type in complex type >

[jira] [Commented] (SPARK-32877) Fix Hive UDF not support decimal type in complex type

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195365#comment-17195365 ] Apache Spark commented on SPARK-32877: -- User 'ulysses-you' has created a pull reque

[jira] [Resolved] (SPARK-32868) Mark more aggregate functions as order irrelevant

2020-09-14 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32868. -- Fix Version/s: 3.1.0 Assignee: Tanel Kiis Resolution: Fixed Resolved b

[jira] [Resolved] (SPARK-32780) Fill since fields for all the expressions

2020-09-14 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32780. -- Resolution: Duplicate > Fill since fields for all the expressions > --

[jira] [Created] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread wuyi (Jira)
wuyi created SPARK-32878: Summary: Avoid scheduling TaskSetManager which has no pending tasks Key: SPARK-32878 URL: https://issues.apache.org/jira/browse/SPARK-32878 Project: Spark Issue Type: Improv

[jira] [Created] (SPARK-32879) SessionStateBuilder should apply options set when creating the session state

2020-09-14 Thread Jira
Herman van Hövell created SPARK-32879: - Summary: SessionStateBuilder should apply options set when creating the session state Key: SPARK-32879 URL: https://issues.apache.org/jira/browse/SPARK-32879

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-09-14 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195531#comment-17195531 ] Thomas Graves commented on SPARK-32037: --- it is a good point about blocklist being

[jira] [Assigned] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32878: Assignee: (was: Apache Spark) > Avoid scheduling TaskSetManager which has no pending

[jira] [Commented] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195532#comment-17195532 ] Apache Spark commented on SPARK-32878: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32878: Assignee: Apache Spark > Avoid scheduling TaskSetManager which has no pending tasks > ---

[jira] [Commented] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195533#comment-17195533 ] Apache Spark commented on SPARK-32878: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

2020-09-14 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195542#comment-17195542 ] Erik Krogen commented on SPARK-32037: - Thanks for continuing to push this forward [~

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

2020-09-14 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-32708: --- Affects Version/s: (was: 2.4.6) > Query optimization fails to reuse exchange with DataSo

[jira] [Created] (SPARK-32880) Improve and refactor parallel listing in HadoopFSUtils

2020-09-14 Thread Chao Sun (Jira)
Chao Sun created SPARK-32880: Summary: Improve and refactor parallel listing in HadoopFSUtils Key: SPARK-32880 URL: https://issues.apache.org/jira/browse/SPARK-32880 Project: Spark Issue Type: Im

[jira] [Created] (SPARK-32881) NoSuchElementException occurs during decommissioning

2020-09-14 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-32881: - Summary: NoSuchElementException occurs during decommissioning Key: SPARK-32881 URL: https://issues.apache.org/jira/browse/SPARK-32881 Project: Spark Issue

[jira] [Updated] (SPARK-32881) NoSuchElementException occurs during decommissioning

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32881: -- Description: `BlockManagerMasterEndpoint` seems to fail at `getReplicateInfoForRDDBlocks` due

[jira] [Commented] (SPARK-32881) NoSuchElementException occurs during decommissioning

2020-09-14 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195700#comment-17195700 ] Holden Karau commented on SPARK-32881: -- Thanks for the catch, I'll take a look at t

[jira] [Assigned] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32876: - Assignee: Hyukjin Kwon > Change default fallback versions in HiveExternalCatalogVersion

[jira] [Resolved] (SPARK-32876) Change default fallback versions in HiveExternalCatalogVersionsSuite

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32876. --- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pul

[jira] [Resolved] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32872. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution: Fix

[jira] [Created] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread William Hyun (Jira)
William Hyun created SPARK-32882: Summary: Remove python2 installation in K8s python image Key: SPARK-32882 URL: https://issues.apache.org/jira/browse/SPARK-32882 Project: Spark Issue Type: I

[jira] [Assigned] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32882: Assignee: (was: Apache Spark) > Remove python2 installation in K8s python image > ---

[jira] [Commented] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195738#comment-17195738 ] Apache Spark commented on SPARK-32882: -- User 'williamhyun' has created a pull reque

[jira] [Assigned] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32882: Assignee: Apache Spark > Remove python2 installation in K8s python image > --

[jira] [Assigned] (SPARK-32879) SessionStateBuilder should apply options set when creating the session state

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32879: Assignee: Apache Spark (was: Herman van Hövell) > SessionStateBuilder should apply optio

[jira] [Assigned] (SPARK-32879) SessionStateBuilder should apply options set when creating the session state

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32879: Assignee: Herman van Hövell (was: Apache Spark) > SessionStateBuilder should apply optio

[jira] [Commented] (SPARK-32879) SessionStateBuilder should apply options set when creating the session state

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195740#comment-17195740 ] Apache Spark commented on SPARK-32879: -- User 'hvanhovell' has created a pull reques

[jira] [Commented] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195746#comment-17195746 ] Apache Spark commented on SPARK-32872: -- User 'ankurdave' has created a pull request

[jira] [Commented] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195749#comment-17195749 ] Apache Spark commented on SPARK-32872: -- User 'ankurdave' has created a pull request

[jira] [Assigned] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32882: - Assignee: William Hyun > Remove python2 installation in K8s python image >

[jira] [Resolved] (SPARK-32882) Remove python2 installation in K8s python image

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32882. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29751 [https://

[jira] [Resolved] (SPARK-32871) Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32871. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29742 [https://gi

[jira] [Commented] (SPARK-32862) Left semi stream-stream join

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195790#comment-17195790 ] Hyukjin Kwon commented on SPARK-32862: -- Nice. [~chengsu], it would have been better

[jira] [Updated] (SPARK-32834) from_avro is giving empty result

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32834: - Description: I am trying to read a Kafka topic with Spark readStream but getting problem while

[jira] [Updated] (SPARK-32834) from_avro is giving empty result

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32834: - Description: I am trying to read a Kafka topic with Spark readStream but getting problem while

[jira] [Commented] (SPARK-32821) cannot group by with window in sql statement for structured streaming with watermark

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195792#comment-17195792 ] Hyukjin Kwon commented on SPARK-32821: -- [~johnny bai] can you provide a self-contai

[jira] [Updated] (SPARK-32821) cannot group by with window in sql statement for structured streaming with watermark

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32821: - Description: current only support dsl style as below:  {code} import spark.implicits._ val word

[jira] [Resolved] (SPARK-32715) Broadcast block pieces may memory leak

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32715. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution: Fix

[jira] [Assigned] (SPARK-32715) Broadcast block pieces may memory leak

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32715: - Assignee: Lantao Jin > Broadcast block pieces may memory leak > ---

[jira] [Created] (SPARK-32883) Stream-stream join improvement

2020-09-14 Thread Cheng Su (Jira)
Cheng Su created SPARK-32883: Summary: Stream-stream join improvement Key: SPARK-32883 URL: https://issues.apache.org/jira/browse/SPARK-32883 Project: Spark Issue Type: Umbrella Compone

[jira] [Updated] (SPARK-32862) Left semi stream-stream join

2020-09-14 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-32862: - Parent: SPARK-32883 Issue Type: Sub-task (was: New Feature) > Left semi stream-stream join > --

[jira] [Updated] (SPARK-32854) Avoid per-row join type check in stream-stream join

2020-09-14 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-32854: - Parent: SPARK-32883 Issue Type: Sub-task (was: Improvement) > Avoid per-row join type check in

[jira] [Updated] (SPARK-32863) Full outer stream-stream join

2020-09-14 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-32863: - Parent: SPARK-32883 Issue Type: Sub-task (was: New Feature) > Full outer stream-stream join > -

[jira] [Commented] (SPARK-32862) Left semi stream-stream join

2020-09-14 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195802#comment-17195802 ] Cheng Su commented on SPARK-32862: -- Thanks [~hyukjin.kwon], I created an umbrella Jira

[jira] [Commented] (SPARK-32454) Fix HistoryServerSuite in Scala 2.13

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195847#comment-17195847 ] Dongjoon Hyun commented on SPARK-32454: --- Please close this if you check it~ This s

[jira] [Assigned] (SPARK-32875) TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method.

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32875: Assignee: (was: Apache Spark) > TaskSchedulerImplSuite: For the pattern of submitTask

[jira] [Commented] (SPARK-32875) TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method.

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195857#comment-17195857 ] Apache Spark commented on SPARK-32875: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-32875) TaskSchedulerImplSuite: For the pattern of submitTasks + resourceOffers + assert, extract the general method.

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32875: Assignee: Apache Spark > TaskSchedulerImplSuite: For the pattern of submitTasks + resourc

[jira] [Resolved] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32878. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29750 [https://

[jira] [Assigned] (SPARK-32878) Avoid scheduling TaskSetManager which has no pending tasks

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32878: - Assignee: wuyi > Avoid scheduling TaskSetManager which has no pending tasks > -

[jira] [Created] (SPARK-32884) Mark TPCDSQuery*Suite as ExtendedSQLTest

2020-09-14 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-32884: - Summary: Mark TPCDSQuery*Suite as ExtendedSQLTest Key: SPARK-32884 URL: https://issues.apache.org/jira/browse/SPARK-32884 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-32884) Mark TPCDSQuery*Suite as ExtendedSQLTest

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195864#comment-17195864 ] Apache Spark commented on SPARK-32884: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-32884) Mark TPCDSQuery*Suite as ExtendedSQLTest

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32884: Assignee: Apache Spark > Mark TPCDSQuery*Suite as ExtendedSQLTest > -

[jira] [Assigned] (SPARK-32884) Mark TPCDSQuery*Suite as ExtendedSQLTest

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32884: Assignee: (was: Apache Spark) > Mark TPCDSQuery*Suite as ExtendedSQLTest > --

[jira] [Commented] (SPARK-32884) Mark TPCDSQuery*Suite as ExtendedSQLTest

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195865#comment-17195865 ] Apache Spark commented on SPARK-32884: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-32885) Add DataStreamReader.table API

2020-09-14 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-32885: --- Summary: Add DataStreamReader.table API Key: SPARK-32885 URL: https://issues.apache.org/jira/browse/SPARK-32885 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-32879) SessionStateBuilder should apply options set when creating the session state

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32879. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29752 [https://gith

[jira] [Assigned] (SPARK-32885) Add DataStreamReader.table API

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32885: Assignee: Apache Spark > Add DataStreamReader.table API > --

[jira] [Assigned] (SPARK-32885) Add DataStreamReader.table API

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32885: Assignee: (was: Apache Spark) > Add DataStreamReader.table API >

[jira] [Commented] (SPARK-32885) Add DataStreamReader.table API

2020-09-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195904#comment-17195904 ] Apache Spark commented on SPARK-32885: -- User 'xuanyuanking' has created a pull requ

[jira] [Resolved] (SPARK-32454) Fix HistoryServerSuite in Scala 2.13

2020-09-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32454. -- Resolution: Duplicate > Fix HistoryServerSuite in Scala 2.13 > ---

[jira] [Commented] (SPARK-32872) BytesToBytesMap at MAX_CAPACITY exceeds growth threshold

2020-09-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195907#comment-17195907 ] Dongjoon Hyun commented on SPARK-32872: --- Thank you for updating, [~ankurd]! > Byt

[jira] [Resolved] (SPARK-32738) thread safe endpoints may hang due to fatal error

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32738. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29580 [https://gith

[jira] [Assigned] (SPARK-32738) thread safe endpoints may hang due to fatal error

2020-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32738: --- Assignee: Zhenhua Wang > thread safe endpoints may hang due to fatal error > --