[jira] [Updated] (SPARK-34067) PartitionPruning push down pruningHasBenefit function into insertPredicate function to decrease calculate time

2021-01-11 Thread jiahong.li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiahong.li updated SPARK-34067: --- Priority: Minor (was: Trivial) > PartitionPruning push down pruningHasBenefit function into insertP

[jira] [Updated] (SPARK-31952) The metric of MemoryBytesSpill is incorrect when doing Aggregate

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31952: -- Fix Version/s: 3.0.2 > The metric of MemoryBytesSpill is incorrect when doing Aggregate >

[jira] [Assigned] (SPARK-31952) The metric of MemoryBytesSpill is incorrect when doing Aggregate

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31952: - Assignee: wuyi > The metric of MemoryBytesSpill is incorrect when doing Aggregate > ---

[jira] [Commented] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263085#comment-17263085 ] Apache Spark commented on SPARK-34082: -- User 'LantaoJin' has created a pull request

[jira] [Commented] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263084#comment-17263084 ] Apache Spark commented on SPARK-34082: -- User 'LantaoJin' has created a pull request

[jira] [Assigned] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34082: Assignee: Apache Spark > Window expression with alias inside WHERE and HAVING clauses fai

[jira] [Assigned] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34082: Assignee: (was: Apache Spark) > Window expression with alias inside WHERE and HAVING

[jira] [Created] (SPARK-34082) Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions

2021-01-11 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-34082: -- Summary: Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions Key: SPARK-34082 URL: https://issues.apache.org/jira/browse/SPARK-34082

[jira] [Commented] (SPARK-24081) Spark SQL drops the table while writing into table in "overwrite" mode.

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263082#comment-17263082 ] Apache Spark commented on SPARK-24081: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-33690) Escape meta-characters in showString

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263076#comment-17263076 ] Apache Spark commented on SPARK-33690: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-28646) Allow usage of `count` only for parameterless aggregate function

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263075#comment-17263075 ] Apache Spark commented on SPARK-28646: -- User 'gengliangwang' has created a pull req

[jira] [Commented] (SPARK-28646) Allow usage of `count` only for parameterless aggregate function

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263074#comment-17263074 ] Apache Spark commented on SPARK-28646: -- User 'gengliangwang' has created a pull req

[jira] [Updated] (SPARK-34081) Should not pushdown LeftSemi/LeftAnti over Aggregate

2021-01-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34081: Description: {code:scala} spark.range(5).selectExpr("id as a", "id as b").write.saveAsTable("t1")

[jira] [Updated] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34059: - Fix Version/s: 3.1.1 3.0.2 2.4.8 > Use for/foreach rather

[jira] [Created] (SPARK-34081) Should not pushdown LeftSemi/LeftAnti over Aggregate

2021-01-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34081: --- Summary: Should not pushdown LeftSemi/LeftAnti over Aggregate Key: SPARK-34081 URL: https://issues.apache.org/jira/browse/SPARK-34081 Project: Spark Issue Type

[jira] [Commented] (SPARK-33346) Change the never changed var to val

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263065#comment-17263065 ] Apache Spark commented on SPARK-33346: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-33346) Change the never changed var to val

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33346: Assignee: Apache Spark > Change the never changed var to val > --

[jira] [Assigned] (SPARK-33346) Change the never changed var to val

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33346: Assignee: (was: Apache Spark) > Change the never changed var to val > ---

[jira] [Commented] (SPARK-33346) Change the never changed var to val

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263064#comment-17263064 ] Apache Spark commented on SPARK-33346: -- User 'LuciferYang' has created a pull reque

[jira] [Updated] (SPARK-31952) The metric of MemoryBytesSpill is incorrect when doing Aggregate

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31952: -- Issue Type: Bug (was: Improvement) > The metric of MemoryBytesSpill is incorrect when doing A

[jira] [Commented] (SPARK-27981) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263054#comment-17263054 ] Dongjoon Hyun commented on SPARK-27981: --- Yes, it does, but Apache Spark community

[jira] [Comment Edited] (SPARK-27981) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`

2021-01-11 Thread Dustin Smith (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263052#comment-17263052 ] Dustin Smith edited comment on SPARK-27981 at 1/12/21, 3:33 AM: --

[jira] [Commented] (SPARK-27981) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`

2021-01-11 Thread Dustin Smith (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263052#comment-17263052 ] Dustin Smith commented on SPARK-27981: -- This problem exist on 2.4.6 as well > Remo

[jira] [Assigned] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34074: --- Assignee: Maxim Gekk > Update stats when table size changes > -

[jira] [Resolved] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34074. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31135 [https://gith

[jira] [Commented] (SPARK-34078) Provide async variants for Dataset APIs

2021-01-11 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263044#comment-17263044 ] Xiao Li commented on SPARK-34078: - A sample PR? or a lightweight design doc? > Provide

[jira] [Commented] (SPARK-31952) The metric of MemoryBytesSpill is incorrect when doing Aggregate

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263040#comment-17263040 ] Apache Spark commented on SPARK-31952: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-11 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-34045: Assignee: zhengruifeng > OneVsRestModel.transform should not call setter of submodels > -

[jira] [Resolved] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-11 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-34045. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31086 [https://gi

[jira] [Resolved] (SPARK-34066) Fix misleading example about func `JsonToStructs`

2021-01-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-34066. --- Resolution: Not A Problem > Fix misleading example about func `JsonToStructs` >

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263010#comment-17263010 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263008#comment-17263008 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263006#comment-17263006 ] Xiangrui Meng commented on SPARK-34080: --- Not sure if we have time for 3.1.1 releas

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-34080: -- Affects Version/s: 3.1.1 > Add UnivariateFeatureSelector to deprecate existing selectors > ---

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-34080: -- Description: In SPARK-26111, we introduced a few univariate feature selectors, which share a

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-34080: -- Description: In SPARK-26111, we introduced a few univariate feature selectors, which share a

[jira] [Updated] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-34080: -- Description: In SPARK-26111, we introduced a few univariate feature selectors, which share a

[jira] [Created] (SPARK-34080) Add UnivariateFeatureSelector to deprecate existing selectors

2021-01-11 Thread Xiangrui Meng (Jira)
Xiangrui Meng created SPARK-34080: - Summary: Add UnivariateFeatureSelector to deprecate existing selectors Key: SPARK-34080 URL: https://issues.apache.org/jira/browse/SPARK-34080 Project: Spark

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263004#comment-17263004 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263003#comment-17263003 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262998#comment-17262998 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-34059) Use for/foreach rather than map to make sure execute it eagerly

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262996#comment-17262996 ] Apache Spark commented on SPARK-34059: -- User 'HyukjinKwon' has created a pull reque

[jira] [Updated] (SPARK-34079) Improvement CTE table scan

2021-01-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34079: Description: Prepare table: {code:sql} CREATE TABLE store_sales ( ss_sold_date_sk INT, ss_sold_t

[jira] [Commented] (SPARK-34079) Improvement CTE table scan

2021-01-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262995#comment-17262995 ] Yuming Wang commented on SPARK-34079: - cc [~petertoth] > Improvement CTE table scan

[jira] [Updated] (SPARK-34079) Improvement CTE table scan

2021-01-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34079: Description: Prepare table: {code:sql} CREATE TABLE store_sales ( ss_sold_date_sk INT, ss_sold_t

[jira] [Created] (SPARK-34079) Improvement CTE table scan

2021-01-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34079: --- Summary: Improvement CTE table scan Key: SPARK-34079 URL: https://issues.apache.org/jira/browse/SPARK-34079 Project: Spark Issue Type: Improvement Co

[jira] [Commented] (SPARK-34078) Provide async variants for Dataset APIs

2021-01-11 Thread Yesheng Ma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262984#comment-17262984 ] Yesheng Ma commented on SPARK-34078: [~cloud_fan] [~smilegator] Could you shed some

[jira] [Created] (SPARK-34078) Provide async variants for Dataset APIs

2021-01-11 Thread Yesheng Ma (Jira)
Yesheng Ma created SPARK-34078: -- Summary: Provide async variants for Dataset APIs Key: SPARK-34078 URL: https://issues.apache.org/jira/browse/SPARK-34078 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262981#comment-17262981 ] Apache Spark commented on SPARK-34076: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34076: Assignee: Apache Spark > SQLContext.dropTempTable fails if cache is non-empty > -

[jira] [Commented] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262980#comment-17262980 ] Apache Spark commented on SPARK-34076: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34076: Assignee: (was: Apache Spark) > SQLContext.dropTempTable fails if cache is non-empty

[jira] [Updated] (SPARK-34077) Metastore daemon threads not cleaned up with session.stop()

2021-01-11 Thread Adam Hamlin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Hamlin updated SPARK-34077: Description: Basically, spinning down a SparkSession using session.stop() does not appear to clea

[jira] [Updated] (SPARK-34077) Metastore daemon threads not cleaned up with session.stop()

2021-01-11 Thread Adam Hamlin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Hamlin updated SPARK-34077: Priority: Major (was: Minor) > Metastore daemon threads not cleaned up with session.stop() >

[jira] [Created] (SPARK-34077) Metastore daemon threads not cleaned up with session.stop()

2021-01-11 Thread Adam Hamlin (Jira)
Adam Hamlin created SPARK-34077: --- Summary: Metastore daemon threads not cleaned up with session.stop() Key: SPARK-34077 URL: https://issues.apache.org/jira/browse/SPARK-34077 Project: Spark Is

[jira] [Resolved] (SPARK-33711) Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns

2021-01-11 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33711. -- Fix Version/s: 3.1.1 3.2.0 Assignee: Attila Zsolt Piros Reso

[jira] [Updated] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34076: - Affects Version/s: (was: 3.0.1) 3.1.0 > SQLContext.dropTempTable fails if cac

[jira] [Created] (SPARK-34076) SQLContext.dropTempTable fails if cache is non-empty

2021-01-11 Thread Chao Sun (Jira)
Chao Sun created SPARK-34076: Summary: SQLContext.dropTempTable fails if cache is non-empty Key: SPARK-34076 URL: https://issues.apache.org/jira/browse/SPARK-34076 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262911#comment-17262911 ] Apache Spark commented on SPARK-34074: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34074: Assignee: (was: Apache Spark) > Update stats when table size changes > --

[jira] [Commented] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262909#comment-17262909 ] Apache Spark commented on SPARK-34074: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34074: Assignee: Apache Spark > Update stats when table size changes > -

[jira] [Created] (SPARK-34075) Hidden directories are being listed for partition inference

2021-01-11 Thread Burak Yavuz (Jira)
Burak Yavuz created SPARK-34075: --- Summary: Hidden directories are being listed for partition inference Key: SPARK-34075 URL: https://issues.apache.org/jira/browse/SPARK-34075 Project: Spark Is

[jira] [Updated] (SPARK-33970) Check isNull and isNotNull in tests

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33970: -- Fix Version/s: (was: 3.1.0) 3.1.1 > Check isNull and isNotNull in tests

[jira] [Updated] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34002: -- Fix Version/s: (was: 3.1.0) 3.1.1 > Broken UDF Encoding > -

[jira] [Resolved] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34002. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 31103 [https://

[jira] [Assigned] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34002: - Assignee: L. C. Hsieh > Broken UDF Encoding > --- > > K

[jira] [Commented] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262889#comment-17262889 ] Dongjoon Hyun commented on SPARK-34002: --- I marked this as a blocker for Apache Spa

[jira] [Updated] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34002: -- Target Version/s: 3.1.1 > Broken UDF Encoding > --- > > Key: S

[jira] [Updated] (SPARK-34002) Broken UDF Encoding

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34002: -- Priority: Blocker (was: Major) > Broken UDF Encoding > --- > >

[jira] [Commented] (SPARK-33970) Check isNull and isNotNull in tests

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262860#comment-17262860 ] Apache Spark commented on SPARK-33970: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-34074) Update stats when table size changes

2021-01-11 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-34074: -- Summary: Update stats when table size changes Key: SPARK-34074 URL: https://issues.apache.org/jira/browse/SPARK-34074 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262836#comment-17262836 ] Dongjoon Hyun commented on SPARK-33507: --- Sure, no problem, [~csun]. > Improve and

[jira] [Assigned] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33507: - Assignee: (was: Chao Sun) > Improve and fix cache behavior in v1 and v2 > -

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262826#comment-17262826 ] Chao Sun commented on SPARK-33507: -- [~dongjoon], [~hyukjin.kwon]: can either of you rem

[jira] [Resolved] (SPARK-33991) Repair enumeration conversion error for AllJobsPage

2021-01-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-33991. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31015 [https://gi

[jira] [Updated] (SPARK-34073) cipher problem with IBM JDK

2021-01-11 Thread Peter Heuchert (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Heuchert updated SPARK-34073: --- Attachment: config > cipher problem with IBM JDK > --- > >

[jira] [Assigned] (SPARK-33991) Repair enumeration conversion error for AllJobsPage

2021-01-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-33991: Assignee: yikf > Repair enumeration conversion error for AllJobsPage > --

[jira] [Updated] (SPARK-33991) Repair enumeration conversion error for AllJobsPage

2021-01-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-33991: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Critical) > Repair enumeration

[jira] [Assigned] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26836: Assignee: (was: Apache Spark) > Columns get switched in Spark SQL using Avro backed H

[jira] [Updated] (SPARK-34060) ALTER TABLE .. DROP PARTITION uncaches Hive table while updating table stats

2021-01-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34060: Fix Version/s: 3.1.1 3.0.2 > ALTER TABLE .. DROP PARTITION uncaches Hive table

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262681#comment-17262681 ] Apache Spark commented on SPARK-26836: -- User 'attilapiros' has created a pull reque

[jira] [Assigned] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26836: Assignee: Apache Spark > Columns get switched in Spark SQL using Avro backed Hive table i

[jira] [Created] (SPARK-34073) cipher problem with IBM JDK

2021-01-11 Thread Peter Heuchert (Jira)
Peter Heuchert created SPARK-34073: -- Summary: cipher problem with IBM JDK Key: SPARK-34073 URL: https://issues.apache.org/jira/browse/SPARK-34073 Project: Spark Issue Type: Bug Com

[jira] [Commented] (SPARK-34072) Fix empty array failed in functions.lit()

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262653#comment-17262653 ] Apache Spark commented on SPARK-34072: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-34072) Fix empty array failed in functions.lit()

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262651#comment-17262651 ] Apache Spark commented on SPARK-34072: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-34072) Fix empty array failed in functions.lit()

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34072: Assignee: Apache Spark > Fix empty array failed in functions.lit() >

[jira] [Assigned] (SPARK-34072) Fix empty array failed in functions.lit()

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34072: Assignee: (was: Apache Spark) > Fix empty array failed in functions.lit() > -

[jira] [Created] (SPARK-34072) Fix empty array failed in functions.lit()

2021-01-11 Thread ulysses you (Jira)
ulysses you created SPARK-34072: --- Summary: Fix empty array failed in functions.lit() Key: SPARK-34072 URL: https://issues.apache.org/jira/browse/SPARK-34072 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34071: Assignee: Apache Spark > Check stats of cached table after altering > --

[jira] [Assigned] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34071: Assignee: Apache Spark > Check stats of cached table after altering > --

[jira] [Commented] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262642#comment-17262642 ] Apache Spark commented on SPARK-34071: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34071: Assignee: (was: Apache Spark) > Check stats of cached table after altering > ---

[jira] [Updated] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-34071: --- Description: 1. Port the test added by https://github.com/apache/spark/pull/31112 for `ALTER TABLE

[jira] [Created] (SPARK-34071) Check stats of cached table after altering

2021-01-11 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-34071: -- Summary: Check stats of cached table after altering Key: SPARK-34071 URL: https://issues.apache.org/jira/browse/SPARK-34071 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-34070) Replaces find and emptiness check with exists.

2021-01-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-34070: - Summary: Replaces find and emptiness check with exists. (was: Use exists method to simplify 'find + emp

[jira] [Commented] (SPARK-34070) Use exists method to simplify 'find + emptiness check' on Seq

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262585#comment-17262585 ] Apache Spark commented on SPARK-34070: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-34070) Use exists method to simplify 'find + emptiness check' on Seq

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34070: Assignee: (was: Apache Spark) > Use exists method to simplify 'find + emptiness check

[jira] [Commented] (SPARK-34070) Use exists method to simplify 'find + emptiness check' on Seq

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262587#comment-17262587 ] Apache Spark commented on SPARK-34070: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-34070) Use exists method to simplify 'find + emptiness check' on Seq

2021-01-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34070: Assignee: Apache Spark > Use exists method to simplify 'find + emptiness check' on Seq >

  1   2   >