[jira] [Created] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Rui Wang (Jira)
Rui Wang created SPARK-40971: Summary: Imports more from connect proto package to avoid calling `proto.` for Connect DSL Key: SPARK-40971 URL: https://issues.apache.org/jira/browse/SPARK-40971 Project: Sp

[jira] [Commented] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626440#comment-17626440 ] Apache Spark commented on SPARK-40971: -- User 'amaliujia' has created a pull request

[jira] [Assigned] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40971: Assignee: Apache Spark > Imports more from connect proto package to avoid calling `proto.

[jira] [Assigned] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40971: Assignee: (was: Apache Spark) > Imports more from connect proto package to avoid call

[jira] [Created] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
Mingming Ge created SPARK-40972: --- Summary: OptimizeLocalShuffleReader causing data skew Key: SPARK-40972 URL: https://issues.apache.org/jira/browse/SPARK-40972 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Description: !image-2022-10-31-15-50-36-435.png! (was: !image-2022-10-31-15-49-36-559.png!) > Op

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Attachment: image-2022-10-31-15-50-36-435.png > OptimizeLocalShuffleReader causing data skew > ---

[jira] [Commented] (SPARK-40973) Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT

2022-10-31 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626449#comment-17626449 ] Haejoon Lee commented on SPARK-40973: - I'm working on it > Rename _LEGACY_ERROR_TEM

[jira] [Created] (SPARK-40973) Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT

2022-10-31 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-40973: --- Summary: Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT Key: SPARK-40973 URL: https://issues.apache.org/jira/browse/SPARK-40973 Project: Spark I

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Attachment: image-2022-10-31-15-51-39-430.png > OptimizeLocalShuffleReader causing data skew > ---

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Description:   !image-2022-10-31-15-53-19-751.png! !image-2022-10-31-15-50-36-435.png!     !i

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Attachment: image-2022-10-31-15-53-19-751.png > OptimizeLocalShuffleReader causing data skew > ---

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Description: Because there are many empty files in the table, the partition num of OptimizeLocalS

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Attachment: image-2022-10-31-15-57-41-599.png > OptimizeLocalShuffleReader causing data skew > ---

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Description: Because there are many empty files in the table, the partition num of OptimizeLocalS

[jira] [Updated] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Mingming Ge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingming Ge updated SPARK-40972: Description: Because there are many empty files in the table, the partition num of OptimizeLocalS

[jira] [Commented] (SPARK-40794) Upgrade Netty from 4.1.80 to 4.1.84

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626455#comment-17626455 ] Apache Spark commented on SPARK-40794: -- User 'clairezhuang' has created a pull requ

[jira] [Assigned] (SPARK-40973) Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40973: Assignee: (was: Apache Spark) > Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_

[jira] [Assigned] (SPARK-40973) Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40973: Assignee: Apache Spark > Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT > -

[jira] [Commented] (SPARK-40973) Rename _LEGACY_ERROR_TEMP_0055 to UNCLOSED_BRACKETED_COMMENT

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626456#comment-17626456 ] Apache Spark commented on SPARK-40973: -- User 'itholic' has created a pull request f

[jira] [Created] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Omar Ismail (Jira)
Omar Ismail created SPARK-40974: --- Summary: EXPODE function selects outer column Key: SPARK-40974 URL: https://issues.apache.org/jira/browse/SPARK-40974 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626489#comment-17626489 ] Apache Spark commented on SPARK-40974: -- User 'clairezhuang' has created a pull requ

[jira] [Assigned] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40974: Assignee: (was: Apache Spark) > EXPODE function selects outer column > --

[jira] [Assigned] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40974: Assignee: Apache Spark > EXPODE function selects outer column > -

[jira] [Commented] (SPARK-40972) OptimizeLocalShuffleReader causing data skew

2022-10-31 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626490#comment-17626490 ] Yuming Wang commented on SPARK-40972: - cc [~michaelzhang-db] > OptimizeLocalShuffle

[jira] [Created] (SPARK-40975) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021

2022-10-31 Thread Max Gekk (Jira)
Max Gekk created SPARK-40975: Summary: Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021 Key: SPARK-40975 URL: https://issues.apache.org/jira/browse/SPARK-40975 Project: Spark Issu

[jira] [Assigned] (SPARK-40975) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40975: Assignee: Apache Spark (was: Max Gekk) > Assign a name to the legacy error class _LEGACY

[jira] [Commented] (SPARK-40975) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626532#comment-17626532 ] Apache Spark commented on SPARK-40975: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-40975) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40975: Assignee: Max Gekk (was: Apache Spark) > Assign a name to the legacy error class _LEGACY

[jira] [Commented] (SPARK-40975) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_0021

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626533#comment-17626533 ] Apache Spark commented on SPARK-40975: -- User 'MaxGekk' has created a pull request f

[jira] [Resolved] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40971. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38445 [https://gith

[jira] [Assigned] (SPARK-40971) Imports more from connect proto package to avoid calling `proto.` for Connect DSL

2022-10-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40971: --- Assignee: Rui Wang > Imports more from connect proto package to avoid calling `proto.` for

[jira] [Commented] (SPARK-40798) Alter partition should verify value

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626556#comment-17626556 ] Apache Spark commented on SPARK-40798: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-40663) Migrate execution errors onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626557#comment-17626557 ] Apache Spark commented on SPARK-40663: -- User 'itholic' has created a pull request f

[jira] [Commented] (SPARK-40663) Migrate execution errors onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626558#comment-17626558 ] Apache Spark commented on SPARK-40663: -- User 'itholic' has created a pull request f

[jira] [Commented] (SPARK-34210) Cannot create a record reader because of a previous error when spark accesses the hive on HBase table

2022-10-31 Thread Mehul Thakkar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626576#comment-17626576 ] Mehul Thakkar commented on SPARK-34210: --- Do you mean we have to download the spark

[jira] [Comment Edited] (SPARK-34210) Cannot create a record reader because of a previous error when spark accesses the hive on HBase table

2022-10-31 Thread Mehul Thakkar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626576#comment-17626576 ] Mehul Thakkar edited comment on SPARK-34210 at 10/31/22 12:55 PM:

[jira] [Created] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Yang Jie (Jira)
Yang Jie created SPARK-40976: Summary: Upgrade sbt to 1.7.3 Key: SPARK-40976 URL: https://issues.apache.org/jira/browse/SPARK-40976 Project: Spark Issue Type: Improvement Components: Bu

[jira] [Assigned] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40976: Assignee: (was: Apache Spark) > Upgrade sbt to 1.7.3 > > >

[jira] [Commented] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626630#comment-17626630 ] Apache Spark commented on SPARK-40976: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40976: Assignee: Apache Spark > Upgrade sbt to 1.7.3 > > >

[jira] [Commented] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626631#comment-17626631 ] Apache Spark commented on SPARK-40976: -- User 'LuciferYang' has created a pull reque

[jira] [Comment Edited] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Nikhil Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626635#comment-17626635 ] Nikhil Sharma edited comment on SPARK-33807 at 10/31/22 3:09 PM: -

[jira] [Commented] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Nikhil Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626635#comment-17626635 ] Nikhil Sharma commented on SPARK-33807: --- Thank you for sharing such good informati

[jira] [Comment Edited] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Nikhil Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626635#comment-17626635 ] Nikhil Sharma edited comment on SPARK-33807 at 10/31/22 3:10 PM: -

[jira] [Comment Edited] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Nikhil Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626635#comment-17626635 ] Nikhil Sharma edited comment on SPARK-33807 at 10/31/22 3:10 PM: -

[jira] [Commented] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626642#comment-17626642 ] Apache Spark commented on SPARK-40974: -- User 'clairezhuang' has created a pull requ

[jira] [Commented] (SPARK-40974) EXPODE function selects outer column

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626644#comment-17626644 ] Apache Spark commented on SPARK-40974: -- User 'clairezhuang' has created a pull requ

[jira] [Updated] (SPARK-40916) udf could not filter null value cause npe

2022-10-31 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-40916: Description: {code:sql} select t22.uid, from ( SELECT code, count(distinct

[jira] [Commented] (SPARK-40802) Enhance JDBC Connector to use PreparedStatement.getMetaData() to resolve schema instead of PreparedStatement.executeQuery()

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626743#comment-17626743 ] Apache Spark commented on SPARK-40802: -- User 'Mingli-Rui' has created a pull reques

[jira] [Commented] (SPARK-40802) Enhance JDBC Connector to use PreparedStatement.getMetaData() to resolve schema instead of PreparedStatement.executeQuery()

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626742#comment-17626742 ] Apache Spark commented on SPARK-40802: -- User 'Mingli-Rui' has created a pull reques

[jira] [Assigned] (SPARK-40802) Enhance JDBC Connector to use PreparedStatement.getMetaData() to resolve schema instead of PreparedStatement.executeQuery()

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40802: Assignee: (was: Apache Spark) > Enhance JDBC Connector to use PreparedStatement.getMe

[jira] [Assigned] (SPARK-40802) Enhance JDBC Connector to use PreparedStatement.getMetaData() to resolve schema instead of PreparedStatement.executeQuery()

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40802: Assignee: Apache Spark > Enhance JDBC Connector to use PreparedStatement.getMetaData() to

[jira] [Commented] (SPARK-40569) Add smoke test in standalone cluster for spark-docker

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626753#comment-17626753 ] Vivek Garg commented on SPARK-40569: The Salesforce Marketing Cloud training offered

[jira] [Comment Edited] (SPARK-40569) Add smoke test in standalone cluster for spark-docker

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626753#comment-17626753 ] Vivek Garg edited comment on SPARK-40569 at 10/31/22 6:38 PM:

[jira] [Comment Edited] (SPARK-40569) Add smoke test in standalone cluster for spark-docker

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626753#comment-17626753 ] Vivek Garg edited comment on SPARK-40569 at 10/31/22 6:38 PM:

[jira] [Comment Edited] (SPARK-40569) Add smoke test in standalone cluster for spark-docker

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626753#comment-17626753 ] Vivek Garg edited comment on SPARK-40569 at 10/31/22 6:39 PM:

[jira] (SPARK-40569) Add smoke test in standalone cluster for spark-docker

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40569 ] Vivek Garg deleted comment on SPARK-40569: was (Author: JIRAUSER294516): The Salesforce Marketing Cloud training offered by IgmGuru is created by instructors who are experts in the field usi

[jira] [Commented] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626759#comment-17626759 ] Vivek Garg commented on SPARK-33807: Thank [you|https://www.igmguru.com/salesforce/

[jira] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22588 ] Vivek Garg deleted comment on SPARK-22588: was (Author: JIRAUSER294516): Thank [you|https://www.igmguru.com/salesforce/salesforce-marketing-cloud-training/]. > SPARK: Load Data from Datafra

[jira] [Commented] (SPARK-22588) SPARK: Load Data from Dataframe or RDD to DynamoDB / dealing with null values

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626758#comment-17626758 ] Vivek Garg commented on SPARK-22588: Thank [you|https://www.igmguru.com/salesforce/

[jira] [Comment Edited] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626759#comment-17626759 ] Vivek Garg edited comment on SPARK-33807 at 10/31/22 6:42 PM:

[jira] [Comment Edited] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626759#comment-17626759 ] Vivek Garg edited comment on SPARK-33807 at 10/31/22 6:43 PM:

[jira] [Commented] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2022-10-31 Thread Vivek Garg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626762#comment-17626762 ] Vivek Garg commented on SPARK-23521: IgmGuru [Mulesoft Online Training|https://www.

[jira] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807 ] Chao Sun deleted comment on SPARK-33807: -- was (Author: JIRAUSER294516): Great job. [Salesforce Marketing Cloud Certification|https://www.igmguru.com/salesforce/salesforce-marketing-cloud-traini

[jira] (SPARK-33807) Data Source V2: Remove read specific distributions

2022-10-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807 ] Chao Sun deleted comment on SPARK-33807: -- was (Author: JIRAUSER295436): Thank you for sharing such good information. Very informative and effective post.  +[https://www.igmguru.com/digital-mar

[jira] [Created] (SPARK-40977) Complete Support for Union in Python client

2022-10-31 Thread Rui Wang (Jira)
Rui Wang created SPARK-40977: Summary: Complete Support for Union in Python client Key: SPARK-40977 URL: https://issues.apache.org/jira/browse/SPARK-40977 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-40977) Complete Support for Union in Python client

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626799#comment-17626799 ] Apache Spark commented on SPARK-40977: -- User 'amaliujia' has created a pull request

[jira] [Commented] (SPARK-40977) Complete Support for Union in Python client

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626800#comment-17626800 ] Apache Spark commented on SPARK-40977: -- User 'amaliujia' has created a pull request

[jira] [Assigned] (SPARK-40977) Complete Support for Union in Python client

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40977: Assignee: Apache Spark > Complete Support for Union in Python client > --

[jira] [Assigned] (SPARK-40977) Complete Support for Union in Python client

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40977: Assignee: (was: Apache Spark) > Complete Support for Union in Python client > ---

[jira] [Assigned] (SPARK-40947) Upgrade pandas to 1.5.1

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40947: - Assignee: Haejoon Lee > Upgrade pandas to 1.5.1 > --- > >

[jira] [Resolved] (SPARK-40947) Upgrade pandas to 1.5.1

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40947. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38420 [https://

[jira] [Assigned] (SPARK-40966) FIX `read_parquet` with `pandas_metadata`

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40966: - Assignee: Haejoon Lee > FIX `read_parquet` with `pandas_metadata` > ---

[jira] [Resolved] (SPARK-40966) FIX `read_parquet` with `pandas_metadata`

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40966. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38420 [https://

[jira] [Resolved] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40976. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38451 [https://

[jira] [Assigned] (SPARK-40976) Upgrade sbt to 1.7.3

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40976: - Assignee: Yang Jie > Upgrade sbt to 1.7.3 > > > Ke

[jira] [Created] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Max Gekk (Jira)
Max Gekk created SPARK-40978: Summary: Migrate failAnalysis() w/o context onto error classes Key: SPARK-40978 URL: https://issues.apache.org/jira/browse/SPARK-40978 Project: Spark Issue Type: Sub

[jira] [Updated] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40978: - Description: Call `failAnalysis()` w/o context but with an error class instead of `failAnalysis()` w/ a

[jira] [Created] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-40979: - Summary: Keep removed executor info in decommission state Key: SPARK-40979 URL: https://issues.apache.org/jira/browse/SPARK-40979 Project: Spark Issue Type

[jira] [Updated] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40979: -- Reporter: Zhongwei Zhu (was: Dongjoon Hyun) > Keep removed executor info in decommission stat

[jira] [Commented] (SPARK-31776) Literal lit() supports lists and numpy arrays

2022-10-31 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626844#comment-17626844 ] Xinrong Meng commented on SPARK-31776: -- `lit` supports Python list and NumPy arrays

[jira] [Assigned] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40979: Assignee: (was: Apache Spark) > Keep removed executor info in decommission state > --

[jira] [Assigned] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40979: Assignee: Apache Spark > Keep removed executor info in decommission state > -

[jira] [Commented] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626845#comment-17626845 ] Apache Spark commented on SPARK-40979: -- User 'warrenzhu25' has created a pull reque

[jira] [Commented] (SPARK-6857) Python SQL schema inference should support numpy types

2022-10-31 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626847#comment-17626847 ] Xinrong Meng commented on SPARK-6857: - Hi, we have NumPy input support https://issue

[jira] [Commented] (SPARK-37697) Make it easier to convert numpy arrays to Spark Dataframes

2022-10-31 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626850#comment-17626850 ] Xinrong Meng commented on SPARK-37697: -- Hi, we have NumPy input support  https://is

[jira] [Updated] (SPARK-40979) Keep removed executor info in decommission state

2022-10-31 Thread Zhongwei Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhongwei Zhu updated SPARK-40979: - Description: Removed executor due to decommission should be kept in a separate set. To avoid OO

[jira] [Assigned] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40978: Assignee: Max Gekk (was: Apache Spark) > Migrate failAnalysis() w/o context onto error c

[jira] [Commented] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626853#comment-17626853 ] Apache Spark commented on SPARK-40978: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40978: Assignee: Apache Spark (was: Max Gekk) > Migrate failAnalysis() w/o context onto error c

[jira] [Commented] (SPARK-40978) Migrate failAnalysis() w/o context onto error classes

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626854#comment-17626854 ] Apache Spark commented on SPARK-40978: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-37946) Use error classes in the execution errors related to partitions

2022-10-31 Thread Khalid Mammadov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626866#comment-17626866 ] Khalid Mammadov commented on SPARK-37946: - Hi [~maxgekk], I see this one is not

[jira] [Resolved] (SPARK-40815) SymlinkTextInputFormat returns incorrect result due to enabled spark.hadoopRDD.ignoreEmptySplits

2022-10-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40815. --- Fix Version/s: 3.4.0 Assignee: Ivan Sadikov Resolution: Fixed This is resolv

[jira] [Commented] (SPARK-40951) pyspark-connect tests should be skipped if pandas doesn't exist

2022-10-31 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626878#comment-17626878 ] Rui Wang commented on SPARK-40951: -- [~dongjoon] Is this JIRA fully resolved already? Ca

[jira] [Assigned] (SPARK-40944) Relax ordering constraint for CREATE TABLE column options

2022-10-31 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-40944: -- Assignee: Daniel > Relax ordering constraint for CREATE TABLE column options > --

[jira] [Resolved] (SPARK-40944) Relax ordering constraint for CREATE TABLE column options

2022-10-31 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-40944. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38418 [https:

[jira] [Commented] (SPARK-29683) Job failed due to executor failures all available nodes are blacklisted

2022-10-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626884#comment-17626884 ] Attila Zsolt Piros commented on SPARK-29683: [~srowen] I think we can close

[jira] [Updated] (SPARK-40933) Reimplement df.stat.{cov, corr} with built-in sql functions

2022-10-31 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-40933: -- Summary: Reimplement df.stat.{cov, corr} with built-in sql functions (was: Make df.stat.{cov,

[jira] [Assigned] (SPARK-40827) Re-enable the DataFrame.corrwith test after fixing in future pandas.

2022-10-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40827: Assignee: Apache Spark > Re-enable the DataFrame.corrwith test after fixing in future pan

  1   2   >