[jira] [Commented] (SPARK-39159) Add new Dataset API for Offset

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535908#comment-17535908 ] Apache Spark commented on SPARK-39159: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-39159) Add new Dataset API for Offset

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39159: Assignee: (was: Apache Spark) > Add new Dataset API for Offset >

[jira] [Commented] (SPARK-39160) Remove workaround for ARROW-1948

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535907#comment-17535907 ] Apache Spark commented on SPARK-39160: -- User 'pan3793' has created a pull request f

[jira] [Assigned] (SPARK-39159) Add new Dataset API for Offset

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39159: Assignee: Apache Spark > Add new Dataset API for Offset > --

[jira] [Assigned] (SPARK-39160) Remove workaround for ARROW-1948

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39160: Assignee: Apache Spark > Remove workaround for ARROW-1948 > -

[jira] [Assigned] (SPARK-39160) Remove workaround for ARROW-1948

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39160: Assignee: (was: Apache Spark) > Remove workaround for ARROW-1948 > --

[jira] [Created] (SPARK-39160) Remove workaround for ARROW-1948

2022-05-11 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-39160: - Summary: Remove workaround for ARROW-1948 Key: SPARK-39160 URL: https://issues.apache.org/jira/browse/SPARK-39160 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-39086) Support UDT in Spark Parquet vectorized reader

2022-05-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-39086. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36427 [https://github.com

[jira] [Assigned] (SPARK-39086) Support UDT in Spark Parquet vectorized reader

2022-05-11 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-39086: Assignee: Ivan Sadikov > Support UDT in Spark Parquet vectorized reader > ---

[jira] [Created] (SPARK-39159) Add new Dataset API for Offset

2022-05-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-39159: -- Summary: Add new Dataset API for Offset Key: SPARK-39159 URL: https://issues.apache.org/jira/browse/SPARK-39159 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-36837) Upgrade Kafka to 3.1.0

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535880#comment-17535880 ] Apache Spark commented on SPARK-36837: -- User 'dongjoon-hyun' has created a pull req

[jira] [Resolved] (SPARK-37702) Support ANSI Aggregate Function: regr_syy

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37702. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36292 [https://gith

[jira] [Assigned] (SPARK-37702) Support ANSI Aggregate Function: regr_syy

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37702: --- Assignee: jiaan.geng > Support ANSI Aggregate Function: regr_syy >

[jira] [Updated] (SPARK-39060) Typo in error messages of decimal overflow

2022-05-11 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-39060: - Fix Version/s: 3.2.2 > Typo in error messages of decimal overflow >

[jira] [Updated] (SPARK-39158) A valid DECIMAL inserted by DataFrame cannot be read in HiveQL

2022-05-11 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-39158: - Description: h2. Describe the bug We are trying to save a table containing a `{{{}DecimalType{}}}` column cons

[jira] [Updated] (SPARK-39158) A valid DECIMAL inserted by DataFrame cannot be read in HiveQL

2022-05-11 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-39158: - Description: h2. Describe the bug We are trying to save a table containing a `{{{}DecimalType{}}}` column cons

[jira] [Updated] (SPARK-39158) A valid DECIMAL inserted by DataFrame cannot be read in HiveQL

2022-05-11 Thread xsys (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xsys updated SPARK-39158: - Description: h2. Describe the bug We are trying to save a table containing a `DecimalType` column constructed

[jira] [Created] (SPARK-39158) A valid DECIMAL inserted by DataFrame cannot be read in HiveQL

2022-05-11 Thread xsys (Jira)
xsys created SPARK-39158: Summary: A valid DECIMAL inserted by DataFrame cannot be read in HiveQL Key: SPARK-39158 URL: https://issues.apache.org/jira/browse/SPARK-39158 Project: Spark Issue Type: B

[jira] [Assigned] (SPARK-39157) H2Dialect should override getJDBCType so as make the data type is correct

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39157: Assignee: Apache Spark > H2Dialect should override getJDBCType so as make the data type i

[jira] [Assigned] (SPARK-39157) H2Dialect should override getJDBCType so as make the data type is correct

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39157: Assignee: (was: Apache Spark) > H2Dialect should override getJDBCType so as make the

[jira] [Commented] (SPARK-39157) H2Dialect should override getJDBCType so as make the data type is correct

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535868#comment-17535868 ] Apache Spark commented on SPARK-39157: -- User 'beliefer' has created a pull request

[jira] [Created] (SPARK-39157) H2Dialect should override getJDBCType so as make the data type is correct

2022-05-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-39157: -- Summary: H2Dialect should override getJDBCType so as make the data type is correct Key: SPARK-39157 URL: https://issues.apache.org/jira/browse/SPARK-39157 Project: Spark

[jira] [Assigned] (SPARK-39156) Remove ParquetLogRedirector usage from ParquetFileFormat

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39156: Assignee: Apache Spark > Remove ParquetLogRedirector usage from ParquetFileFormat > -

[jira] [Assigned] (SPARK-39156) Remove ParquetLogRedirector usage from ParquetFileFormat

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39156: Assignee: (was: Apache Spark) > Remove ParquetLogRedirector usage from ParquetFileFor

[jira] [Commented] (SPARK-39156) Remove ParquetLogRedirector usage from ParquetFileFormat

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535853#comment-17535853 ] Apache Spark commented on SPARK-39156: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39155: Assignee: Xinrong Meng > Access to JVM through passed-in GatewayClient during type conver

[jira] [Resolved] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39155. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36504 [https://gi

[jira] [Resolved] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39149. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36508 [https://gith

[jira] [Assigned] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39149: --- Assignee: Wenchen Fan > SHOW DATABASES command should not quote database names under legacy

[jira] [Created] (SPARK-39156) Remove ParquetLogRedirector usage from ParquetFileFormat

2022-05-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-39156: Summary: Remove ParquetLogRedirector usage from ParquetFileFormat Key: SPARK-39156 URL: https://issues.apache.org/jira/browse/SPARK-39156 Project: Spark Issue Type:

[jira] [Commented] (SPARK-39078) Support UPDATE commands with DEFAULT values

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535798#comment-17535798 ] Apache Spark commented on SPARK-39078: -- User 'dtenedor' has created a pull request

[jira] [Assigned] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39155: Assignee: Apache Spark > Access to JVM through passed-in GatewayClient during type conver

[jira] [Assigned] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39155: Assignee: (was: Apache Spark) > Access to JVM through passed-in GatewayClient during

[jira] [Commented] (SPARK-39154) Remove outdated statements on distributed-sequence default index

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535796#comment-17535796 ] Apache Spark commented on SPARK-39154: -- User 'xinrong-databricks' has created a pul

[jira] [Commented] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535797#comment-17535797 ] Apache Spark commented on SPARK-39155: -- User 'xinrong-databricks' has created a pul

[jira] [Resolved] (SPARK-39081) Impl DataFrame.resample and Series.resample

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39081. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36420 [https://gi

[jira] [Assigned] (SPARK-39081) Impl DataFrame.resample and Series.resample

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39081: Assignee: zhengruifeng > Impl DataFrame.resample and Series.resample > --

[jira] [Updated] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39155: - Description: Access to JVM through passed-in GatewayClient during type conversion. In customize

[jira] [Created] (SPARK-39155) Access to JVM through passed-in GatewayClient during type conversion

2022-05-11 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-39155: Summary: Access to JVM through passed-in GatewayClient during type conversion Key: SPARK-39155 URL: https://issues.apache.org/jira/browse/SPARK-39155 Project: Spark

[jira] [Assigned] (SPARK-39154) Remove outdated statements on distributed-sequence default index

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39154: Assignee: Xinrong Meng > Remove outdated statements on distributed-sequence default index

[jira] [Resolved] (SPARK-39154) Remove outdated statements on distributed-sequence default index

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39154. -- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-32304) Flaky Test: AdaptiveQueryExecSuite.multiple joins

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32304. -- Resolution: Cannot Reproduce let me resolve this. I don;'t see this anymore > Flaky Test: Ada

[jira] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827 ] Hyukjin Kwon deleted comment on SPARK-34827: -- was (Author: hyukjin.kwon): Uhoh, the JIRA number was wrong. > Support fetching shuffle blocks in batch with i/o encryption > -

[jira] [Reopened] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-34827: -- > Support fetching shuffle blocks in batch with i/o encryption > -

[jira] [Resolved] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34827. -- Resolution: Fixed > Support fetching shuffle blocks in batch with i/o encryption > ---

[jira] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827 ] Hyukjin Kwon deleted comment on SPARK-34827: -- was (Author: apachespark): User 'xinrong-databricks' has created a pull request for this issue: https://github.com/apache/spark/pull/36306 > Su

[jira] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827 ] Hyukjin Kwon deleted comment on SPARK-34827: -- was (Author: apachespark): User 'xinrong-databricks' has created a pull request for this issue: https://github.com/apache/spark/pull/36513 > Su

[jira] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827 ] Hyukjin Kwon deleted comment on SPARK-34827: -- was (Author: apachespark): User 'xinrong-databricks' has created a pull request for this issue: https://github.com/apache/spark/pull/36306 > Su

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535143#comment-17535143 ] Hyukjin Kwon commented on SPARK-34827: -- Uhoh, the JIRA number was wrong. > Support

[jira] [Resolved] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-39147. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36507 [https://gi

[jira] [Assigned] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-39147: Assignee: morvenhuang > Code simplification, use count() instead of filter().size, etc. >

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535091#comment-17535091 ] Apache Spark commented on SPARK-34827: -- User 'xinrong-databricks' has created a pul

[jira] [Created] (SPARK-39154) Remove outdated statements on distributed-sequence default index

2022-05-11 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-39154: Summary: Remove outdated statements on distributed-sequence default index Key: SPARK-39154 URL: https://issues.apache.org/jira/browse/SPARK-39154 Project: Spark

[jira] [Commented] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535088#comment-17535088 ] Apache Spark commented on SPARK-39152: -- User 'attilapiros' has created a pull reque

[jira] [Assigned] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39152: Assignee: Apache Spark (was: Attila Zsolt Piros) > StreamCorruptedException cause job fa

[jira] [Assigned] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39152: Assignee: Attila Zsolt Piros (was: Apache Spark) > StreamCorruptedException cause job fa

[jira] [Assigned] (SPARK-39153) When we look at spark UI or History, we can see the failed tasks first

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39153: Assignee: (was: Apache Spark) > When we look at spark UI or History, we can see the f

[jira] [Commented] (SPARK-39153) When we look at spark UI or History, we can see the failed tasks first

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535086#comment-17535086 ] Apache Spark commented on SPARK-39153: -- User 'smallzhongfeng' has created a pull re

[jira] [Assigned] (SPARK-39153) When we look at spark UI or History, we can see the failed tasks first

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39153: Assignee: Apache Spark > When we look at spark UI or History, we can see the failed tasks

[jira] [Created] (SPARK-39153) When we look at spark UI or History, we can see the failed tasks first

2022-05-11 Thread jingxiong zhong (Jira)
jingxiong zhong created SPARK-39153: --- Summary: When we look at spark UI or History, we can see the failed tasks first Key: SPARK-39153 URL: https://issues.apache.org/jira/browse/SPARK-39153 Project:

[jira] [Created] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-39152: -- Summary: StreamCorruptedException cause job failure for disk persisted RDD Key: SPARK-39152 URL: https://issues.apache.org/jira/browse/SPARK-39152 Project

[jira] [Commented] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535044#comment-17535044 ] Attila Zsolt Piros commented on SPARK-39152: I am working on this > StreamC

[jira] [Commented] (SPARK-39145) CLONE - SPIP: Public APIs for extended Columnar Processing Support

2022-05-11 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535013#comment-17535013 ] Erik Krogen commented on SPARK-39145: - Looks like this is a duplicate of SPARK-27396

[jira] [Commented] (SPARK-36718) only collapse projects if we don't duplicate expensive expressions

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534990#comment-17534990 ] Apache Spark commented on SPARK-36718: -- User 'cloud-fan' has created a pull request

[jira] [Resolved] (SPARK-39079) Catalog name should not contain dot

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39079. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36418 [https://gith

[jira] [Assigned] (SPARK-39079) Catalog name should not contain dot

2022-05-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39079: --- Assignee: Cheng Pan > Catalog name should not contain dot > ---

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2022-05-11 Thread Chris Kimmel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534937#comment-17534937 ] Chris Kimmel commented on SPARK-17556: -- [~jlaskowski] I believe the problem is that

[jira] [Created] (SPARK-39151) Need Support for avro 1.11.0 in apache spark

2022-05-11 Thread Amit Agarwal (Jira)
Amit Agarwal created SPARK-39151: Summary: Need Support for avro 1.11.0 in apache spark Key: SPARK-39151 URL: https://issues.apache.org/jira/browse/SPARK-39151 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534889#comment-17534889 ] Gabor Somogyi commented on SPARK-25355: --- I've had a deeper look and HADOOP_TOKEN_F

[jira] [Commented] (SPARK-39102) Replace the usage of guava's Files.createTempDir() with java.nio.file.Files.createTempDirectory()

2022-05-11 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534870#comment-17534870 ] Yang Jie commented on SPARK-39102: -- Also includes IvyTestUtils and we should add checks

[jira] [Updated] (SPARK-6305) Add support for log4j 2.x to Spark

2022-05-11 Thread prathap nomula (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prathap nomula updated SPARK-6305: -- Attachment: image-2022-05-11-08-33-28-388.png > Add support for log4j 2.x to Spark > --

[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark

2022-05-11 Thread prathap nomula (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534865#comment-17534865 ] prathap nomula commented on SPARK-6305: --- Any updates on the new spark 3.3 release d

[jira] [Updated] (SPARK-38961) Enhance to automatically generate the pandas API support list

2022-05-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38961: - Issue Type: Documentation (was: Test) > Enhance to automatically generate the pandas API suppor

[jira] [Created] (SPARK-39150) Remove `# doctest: +SKIP` of groupby head/tail when infra dump pandas to 1.4+

2022-05-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-39150: --- Summary: Remove `# doctest: +SKIP` of groupby head/tail when infra dump pandas to 1.4+ Key: SPARK-39150 URL: https://issues.apache.org/jira/browse/SPARK-39150 Project:

[jira] [Assigned] (SPARK-38961) Enhance to automatically generate the pandas API support list

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38961: Assignee: Apache Spark > Enhance to automatically generate the pandas API support list >

[jira] [Assigned] (SPARK-38961) Enhance to automatically generate the pandas API support list

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38961: Assignee: (was: Apache Spark) > Enhance to automatically generate the pandas API supp

[jira] [Commented] (SPARK-38961) Enhance to automatically generate the pandas API support list

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534795#comment-17534795 ] Apache Spark commented on SPARK-38961: -- User 'beobest2' has created a pull request

[jira] [Updated] (SPARK-37463) Read/Write Timestamp ntz from/to Orc uses int64

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37463: --- Parent: SPARK-35662 Issue Type: Sub-task (was: Bug) > Read/Write Timestamp ntz from/to Orc

[jira] [Updated] (SPARK-38504) Can't read TimestampNTZ as TimestampLTZ

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38504: --- Parent: SPARK-35662 Issue Type: Sub-task (was: Improvement) > Can't read TimestampNTZ as Ti

[jira] [Assigned] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39149: Assignee: (was: Apache Spark) > SHOW DATABASES command should not quote database name

[jira] [Commented] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534768#comment-17534768 ] Apache Spark commented on SPARK-39149: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39149: Assignee: Apache Spark > SHOW DATABASES command should not quote database names under leg

[jira] [Created] (SPARK-39149) SHOW DATABASES command should not quote database names under legacy mode

2022-05-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-39149: --- Summary: SHOW DATABASES command should not quote database names under legacy mode Key: SPARK-39149 URL: https://issues.apache.org/jira/browse/SPARK-39149 Project: Spark

[jira] [Commented] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534750#comment-17534750 ] Apache Spark commented on SPARK-39147: -- User 'morvenhuang' has created a pull reque

[jira] [Assigned] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39147: Assignee: Apache Spark > Code simplification, use count() instead of filter().size, etc.

[jira] [Assigned] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39147: Assignee: (was: Apache Spark) > Code simplification, use count() instead of filter().

[jira] [Assigned] (SPARK-25050) Handle more than two types in avro union types when writing avro files

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25050: Assignee: (was: Apache Spark) > Handle more than two types in avro union types when w

[jira] [Assigned] (SPARK-25050) Handle more than two types in avro union types when writing avro files

2022-05-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25050: Assignee: Apache Spark > Handle more than two types in avro union types when writing avro

[jira] [Commented] (SPARK-25050) Handle more than two types in avro union types when writing avro files

2022-05-11 Thread Steven Aerts (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534741#comment-17534741 ] Steven Aerts commented on SPARK-25050: -- I created a PR for this issue. Not clear wh

[jira] [Created] (SPARK-39148) Support push down OFFSET append LIMIT to JDBC data source V2

2022-05-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-39148: -- Summary: Support push down OFFSET append LIMIT to JDBC data source V2 Key: SPARK-39148 URL: https://issues.apache.org/jira/browse/SPARK-39148 Project: Spark Iss

[jira] [Updated] (SPARK-38978) Support push down OFFSET to JDBC data source V2

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38978: --- Summary: Support push down OFFSET to JDBC data source V2 (was: Support push down offset to JDBC dat

[jira] [Commented] (SPARK-39148) Support push down OFFSET append LIMIT to JDBC data source V2

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534737#comment-17534737 ] jiaan.geng commented on SPARK-39148: I'm working on. > Support push down OFFSET app

[jira] [Updated] (SPARK-38978) Support push down offset to JDBC data source V2

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38978: --- Description: Currently, DS V2 push-down supports LIMIT but OFFSET. If we can pushing down OFFSET to

[jira] [Updated] (SPARK-38978) Support push down offset to JDBC data source V2

2022-05-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-38978: --- Summary: Support push down offset to JDBC data source V2 (was: Support push down limit and offset t

[jira] [Commented] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread morvenhuang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534732#comment-17534732 ] morvenhuang commented on SPARK-39147: - I'm working on this. > Code simplification,

[jira] [Created] (SPARK-39147) Code simplification, use count() instead of filter().size, etc.

2022-05-11 Thread morvenhuang (Jira)
morvenhuang created SPARK-39147: --- Summary: Code simplification, use count() instead of filter().size, etc. Key: SPARK-39147 URL: https://issues.apache.org/jira/browse/SPARK-39147 Project: Spark