[jira] [Created] (SPARK-34124) Upgrade jackson version to fix CVE-2020-36179 in Spark 2.4

2021-01-14 Thread Yang Jie (Jira)
Yang Jie created SPARK-34124: Summary: Upgrade jackson version to fix CVE-2020-36179 in Spark 2.4 Key: SPARK-34124 URL: https://issues.apache.org/jira/browse/SPARK-34124 Project: Spark Issue

[jira] [Assigned] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34117: Assignee: (was: Apache Spark) > Disable LeftSemi/LeftAnti push down over Aggregate >

[jira] [Commented] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265743#comment-17265743 ] Apache Spark commented on SPARK-34117: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34117: Assignee: Apache Spark > Disable LeftSemi/LeftAnti push down over Aggregate >

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265735#comment-17265735 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34123: Assignee: (was: Apache Spark) > Faster way to display/render entries in HistoryPage

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265731#comment-17265731 ] Jungtaek Lim commented on SPARK-33790: -- Oh OK I haven't encountered the issue but Scala mutable

[jira] [Commented] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265730#comment-17265730 ] Apache Spark commented on SPARK-34123: -- User 'mohan3d' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34123: Assignee: Apache Spark > Faster way to display/render entries in HistoryPage (Spark

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread dzcxzl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265724#comment-17265724 ] dzcxzl commented on SPARK-33790: Thread stack when not working

[jira] [Updated] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Mohanad Elsafty (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohanad Elsafty updated SPARK-34123: Attachment: Screenshot 2021-01-15 at 1.21.40 PM.png > Faster way to display/render

[jira] [Updated] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Mohanad Elsafty (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohanad Elsafty updated SPARK-34123: Description: Since a long time ago my team/company suffered from history server being

[jira] [Created] (SPARK-34123) Faster way to display/render entries in HistoryPage (Spark history server summary page)

2021-01-14 Thread Mohanad Elsafty (Jira)
Mohanad Elsafty created SPARK-34123: --- Summary: Faster way to display/render entries in HistoryPage (Spark history server summary page) Key: SPARK-34123 URL: https://issues.apache.org/jira/browse/SPARK-34123

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265708#comment-17265708 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265707#comment-17265707 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Resolved] (SPARK-32864) Support ORC forced positional evolution

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32864. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 29737

[jira] [Assigned] (SPARK-32864) Support ORC forced positional evolution

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32864: - Assignee: Peter Toth > Support ORC forced positional evolution >

[jira] [Assigned] (SPARK-34122) Remove duplicated branches in case when

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34122: Assignee: Apache Spark > Remove duplicated branches in case when >

[jira] [Assigned] (SPARK-34122) Remove duplicated branches in case when

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34122: Assignee: (was: Apache Spark) > Remove duplicated branches in case when >

[jira] [Commented] (SPARK-34122) Remove duplicated branches in case when

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265702#comment-17265702 ] Apache Spark commented on SPARK-34122: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265700#comment-17265700 ] Jungtaek Lim commented on SPARK-33790: -- {quote} The following is my case 2.x version

[jira] [Created] (SPARK-34122) Remove duplicated branches in case when

2021-01-14 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34122: -- Summary: Remove duplicated branches in case when Key: SPARK-34122 URL: https://issues.apache.org/jira/browse/SPARK-34122 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-34095) Spark 3.0.1 is giving WARN while running job

2021-01-14 Thread Sachit Murarka (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sachit Murarka updated SPARK-34095: --- Priority: Minor (was: Major) > Spark 3.0.1 is giving WARN while running job >

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265697#comment-17265697 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265695#comment-17265695 ] Apache Spark commented on SPARK-33790: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265693#comment-17265693 ] Apache Spark commented on SPARK-33790: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread dzcxzl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265691#comment-17265691 ] dzcxzl commented on SPARK-33790: This is indeed a performance regression problem. The following is my

[jira] [Assigned] (SPARK-34114) should not trim right for read-side char length check and padding

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34114: --- Assignee: Kent Yao > should not trim right for read-side char length check and padding >

[jira] [Resolved] (SPARK-34114) should not trim right for read-side char length check and padding

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34114. - Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31181

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265679#comment-17265679 ] Apache Spark commented on SPARK-33790: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Resolved] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34101. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31173

[jira] [Commented] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265674#comment-17265674 ] Hyukjin Kwon commented on SPARK-12890: -- [~nchammas], shall we file a new JIRA? your description

[jira] [Assigned] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34111: Assignee: (was: Apache Spark) > Deconflict the jars jakarta.servlet-api-4.0.3.jar

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265668#comment-17265668 ] Apache Spark commented on SPARK-34111: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34111: Assignee: Apache Spark > Deconflict the jars jakarta.servlet-api-4.0.3.jar and >

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-14 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265666#comment-17265666 ] Kent Yao commented on SPARK-34111: -- I have found the problem, LOL   In Hadoop 3.2:

[jira] [Commented] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265662#comment-17265662 ] Jungtaek Lim commented on SPARK-33790: -- I've revisited this somehow and I realized this is

[jira] [Created] (SPARK-34121) Intersect operator missing rowCount when CBO enabled

2021-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34121: --- Summary: Intersect operator missing rowCount when CBO enabled Key: SPARK-34121 URL: https://issues.apache.org/jira/browse/SPARK-34121 Project: Spark Issue

[jira] [Assigned] (SPARK-34099) Refactor table caching in `DataSourceV2Strategy`

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34099: --- Assignee: Maxim Gekk > Refactor table caching in `DataSourceV2Strategy` >

[jira] [Resolved] (SPARK-34099) Refactor table caching in `DataSourceV2Strategy`

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34099. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31172

[jira] [Updated] (SPARK-34119) Keep necessary stats after partition pruning

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34119: Parent: SPARK-34120 Issue Type: Sub-task (was: Improvement) > Keep necessary stats after

[jira] [Updated] (SPARK-33956) Add rowCount for Range operator

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33956: Parent: SPARK-34120 Issue Type: Sub-task (was: Improvement) > Add rowCount for Range

[jira] [Updated] (SPARK-33954) Some operator missing rowCount when enable CBO

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33954: Parent: SPARK-34120 Issue Type: Sub-task (was: Improvement) > Some operator missing

[jira] [Updated] (SPARK-34031) Union operator missing rowCount when enable CBO

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34031: Parent: SPARK-34120 Issue Type: Sub-task (was: Improvement) > Union operator missing

[jira] [Updated] (SPARK-33959) Improve the statistics estimation of the Tail

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33959: Parent: SPARK-34120 Issue Type: Sub-task (was: Improvement) > Improve the statistics

[jira] [Created] (SPARK-34120) Improve the statistics estimation

2021-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34120: --- Summary: Improve the statistics estimation Key: SPARK-34120 URL: https://issues.apache.org/jira/browse/SPARK-34120 Project: Spark Issue Type: Umbrella

[jira] [Commented] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265631#comment-17265631 ] Yuming Wang commented on SPARK-34117: - I'm working on. > Disable LeftSemi/LeftAnti push down over

[jira] [Commented] (SPARK-34119) Keep necessary stats after partition pruning

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265657#comment-17265657 ] Yuming Wang commented on SPARK-34119: - [~leoluan] Would you like work on this? > Keep necessary

[jira] [Created] (SPARK-34119) Keep necessary stats after partition pruning

2021-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34119: --- Summary: Keep necessary stats after partition pruning Key: SPARK-34119 URL: https://issues.apache.org/jira/browse/SPARK-34119 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

2021-01-14 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33790: - Priority: Critical (was: Trivial) > Reduce the rpc call of getFileStatus in

[jira] [Updated] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34118: - Fix Version/s: 3.2.0 > Replaces filter and check for emptiness with exists or forall >

[jira] [Updated] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-34118: - Summary: Replaces filter and check for emptiness with exists or forall (was: Replaces filter and check

[jira] [Updated] (SPARK-34118) Replaces filter and check for emptiness with exists or forall.

2021-01-14 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-34118: - Description: Semantic consistency and code Simpilefications before: {code:java} seq.filter(p).size ==

[jira] [Created] (SPARK-34118) Replaces filter and check for emptiness with exists or forall.

2021-01-14 Thread Yang Jie (Jira)
Yang Jie created SPARK-34118: Summary: Replaces filter and check for emptiness with exists or forall. Key: SPARK-34118 URL: https://issues.apache.org/jira/browse/SPARK-34118 Project: Spark

[jira] [Created] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34117: --- Summary: Disable LeftSemi/LeftAnti push down over Aggregate Key: SPARK-34117 URL: https://issues.apache.org/jira/browse/SPARK-34117 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34117: Description: After SPARK-34081, it can improve TPC-DS q38 and q87. But it still can not handle

[jira] [Updated] (SPARK-34117) Disable LeftSemi/LeftAnti push down over Aggregate

2021-01-14 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34117: Attachment: disable_pushdown.jpg current.jpg > Disable LeftSemi/LeftAnti push

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265648#comment-17265648 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34118: Assignee: (was: Apache Spark) > Replaces filter and check for emptiness with exists

[jira] [Assigned] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34118: Assignee: Apache Spark > Replaces filter and check for emptiness with exists or forall >

[jira] [Commented] (SPARK-34118) Replaces filter and check for emptiness with exists or forall

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265647#comment-17265647 ] Apache Spark commented on SPARK-34118: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265611#comment-17265611 ] Hyukjin Kwon commented on SPARK-34080: -- I am okay with having this in Spark 3.1.1. > Add

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34080: - Priority: Critical (was: Major) > Add UnivariateFeatureSelector to deprecate existing

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34080: - Target Version/s: 3.1.1 (was: 3.2.0) > Add UnivariateFeatureSelector to deprecate existing

[jira] [Commented] (SPARK-33711) Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265604#comment-17265604 ] Dongjoon Hyun commented on SPARK-33711: --- +1 for having this in branch-3.0. Could you make a

[jira] [Commented] (SPARK-30747) Update roxygen2 to 7.0.1

2021-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265566#comment-17265566 ] Maciej Szymkiewicz commented on SPARK-30747: Thanks for working on this [~shaneknapp]! >

[jira] [Resolved] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34116. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 31183

[jira] [Updated] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34116: -- Fix Version/s: (was: 3.1.0) 3.1.1 > Separate state store numKeys

[jira] [Updated] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34116: -- Affects Version/s: 3.1.1 > Separate state store numKeys metric test >

[jira] [Commented] (SPARK-33711) Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns

2021-01-14 Thread Shashi Kangayam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265426#comment-17265426 ] Shashi Kangayam commented on SPARK-33711: - [~attilapiros] We have our jobs on Spark-3.0.1 that

[jira] [Resolved] (SPARK-30747) Update roxygen2 to 7.0.1

2021-01-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-30747. - Resolution: Fixed > Update roxygen2 to 7.0.1 > > >

[jira] [Commented] (SPARK-30747) Update roxygen2 to 7.0.1

2021-01-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265142#comment-17265142 ] Shane Knapp commented on SPARK-30747: - we're currently running 7.1.1 on all of the workers. >

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.6

2021-01-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265140#comment-17265140 ] Dongjoon Hyun commented on SPARK-29183: --- Yey! It will be great. Thank you for sharing that,

[jira] [Closed] (SPARK-31693) Investigate AmpLab Jenkins server network issue

2021-01-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp closed SPARK-31693. --- > Investigate AmpLab Jenkins server network issue > --- > >

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.6

2021-01-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265139#comment-17265139 ] Shane Knapp commented on SPARK-29183: - in the next ~month or so, we'll be reimaging the remaining

[jira] [Resolved] (SPARK-31693) Investigate AmpLab Jenkins server network issue

2021-01-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-31693. - Resolution: Fixed > Investigate AmpLab Jenkins server network issue >

[jira] [Assigned] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34116: Assignee: L. C. Hsieh (was: Apache Spark) > Separate state store numKeys metric test >

[jira] [Assigned] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34116: Assignee: Apache Spark (was: L. C. Hsieh) > Separate state store numKeys metric test >

[jira] [Assigned] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34116: Assignee: Apache Spark (was: L. C. Hsieh) > Separate state store numKeys metric test >

[jira] [Commented] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265110#comment-17265110 ] Apache Spark commented on SPARK-34116: -- User 'viirya' has created a pull request for this issue:

[jira] [Created] (SPARK-34116) Separate state store numKeys metric test

2021-01-14 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-34116: --- Summary: Separate state store numKeys metric test Key: SPARK-34116 URL: https://issues.apache.org/jira/browse/SPARK-34116 Project: Spark Issue Type: Test

[jira] [Comment Edited] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2021-01-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253044#comment-17253044 ] Nicholas Chammas edited comment on SPARK-12890 at 1/14/21, 5:41 PM:

[jira] [Commented] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265049#comment-17265049 ] Apache Spark commented on SPARK-34108: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34108: Assignee: (was: Apache Spark) > Caching with permanent view doesn't work in certain

[jira] [Assigned] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34108: Assignee: Apache Spark > Caching with permanent view doesn't work in certain cases >

[jira] [Commented] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265048#comment-17265048 ] Apache Spark commented on SPARK-34108: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-14 Thread Marius Butan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265022#comment-17265022 ] Marius Butan commented on SPARK-34042: -- Like I said in description I made some tests in 

[jira] [Updated] (SPARK-34095) Spark 3.0.1 is giving WARN while running job

2021-01-14 Thread Sachit Murarka (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sachit Murarka updated SPARK-34095: --- Summary: Spark 3.0.1 is giving WARN while running job (was: Spark 3 is giving WARN while

[jira] [Commented] (SPARK-34095) Spark 3 is giving WARN while running job

2021-01-14 Thread Sachit Murarka (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265019#comment-17265019 ] Sachit Murarka commented on SPARK-34095: [~sowen] [~holden] -> Can you please suggest something

[jira] [Issue Comment Deleted] (SPARK-28263) Spark-submit can not find class (ClassNotFoundException)

2021-01-14 Thread Jack LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack LIN updated SPARK-28263: - Comment: was deleted (was: Just guessing, this probably relates to the settings of `sbt`) >

[jira] [Commented] (SPARK-28263) Spark-submit can not find class (ClassNotFoundException)

2021-01-14 Thread Jack LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264990#comment-17264990 ] Jack LIN commented on SPARK-28263: -- Just guessing, this probably relates to the settings of `sbt` >

[jira] [Resolved] (SPARK-33989) Strip auto-generated cast when using Cast.sql

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33989. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31034

[jira] [Assigned] (SPARK-33989) Strip auto-generated cast when using Cast.sql

2021-01-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33989: --- Assignee: ulysses you > Strip auto-generated cast when using Cast.sql >

[jira] [Updated] (SPARK-34102) Spark SQL cannot escape both \ and other special characters

2021-01-14 Thread Noah Kawasaki (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noah Kawasaki updated SPARK-34102: -- Labels: escaping filter string-manipulation (was: ) > Spark SQL cannot escape both \ and

[jira] [Updated] (SPARK-34115) Long runtime on many environment variables

2021-01-14 Thread Norbert Schultz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Schultz updated SPARK-34115: Description: I am not sure if this is a bug report or a feature request. The code is is

[jira] [Updated] (SPARK-34115) Long runtime on many environment variables

2021-01-14 Thread Norbert Schultz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Schultz updated SPARK-34115: Description: I am not sure if this is a bug report or a feature request. The code is is

[jira] [Updated] (SPARK-34115) Long runtime on many environment variables

2021-01-14 Thread Norbert Schultz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Schultz updated SPARK-34115: Description: I am not sure if this is a bug report or a feature request. The code is is

[jira] [Created] (SPARK-34115) Long runtime on many environment variables

2021-01-14 Thread Norbert Schultz (Jira)
Norbert Schultz created SPARK-34115: --- Summary: Long runtime on many environment variables Key: SPARK-34115 URL: https://issues.apache.org/jira/browse/SPARK-34115 Project: Spark Issue Type:

[jira] [Updated] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2021-01-14 Thread jiulongzhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiulongzhu updated SPARK-26385: --- Attachment: (was: SPARK-26385.0001.patch) > YARN - Spark Stateful Structured streaming

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2021-01-14 Thread jiulongzhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264772#comment-17264772 ] jiulongzhu commented on SPARK-26385: https://issues.apache.org/jira/browse/HDFS-9276 > YARN - Spark

  1   2   >