[jira] [Comment Edited] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448412#comment-17448412 ] Jiri Humpolicek edited comment on SPARK-37450 at 11/24/21, 7:39 AM:

[jira] [Commented] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448412#comment-17448412 ] Jiri Humpolicek commented on SPARK-37450: - thanks for answer. Maybe I don't know internals of

[jira] [Updated] (SPARK-37435) Did not find value which can be converted into java.lang.String

[ https://issues.apache.org/jira/browse/SPARK-37435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37435: - Description: Got this following error when loading the saved model. {code} ERROR:ADS Exception

[jira] [Commented] (SPARK-37451) Performance improvement regressed String to Decimal cast

[ https://issues.apache.org/jira/browse/SPARK-37451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448380#comment-17448380 ] Hyukjin Kwon commented on SPARK-37451: -- cc [~yumwang] FYI > Performance improvement regressed

[jira] [Updated] (SPARK-37451) Performance improvement regressed String to Decimal cast

[ https://issues.apache.org/jira/browse/SPARK-37451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37451: - Target Version/s: (was: 3.2.1) > Performance improvement regressed String to Decimal cast >

[jira] [Commented] (SPARK-32446) Add new executor metrics summary REST APIs and parameters

[ https://issues.apache.org/jira/browse/SPARK-32446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448366#comment-17448366 ] Apache Spark commented on SPARK-32446: -- User 'AngersZh' has created a pull request for this

[jira] [Updated] (SPARK-37451) Performance improvement regressed String to Decimal cast

[ https://issues.apache.org/jira/browse/SPARK-37451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raza Jafri updated SPARK-37451: --- Description: A performance improvement to how Spark casts Strings to Decimal in this [PR

[jira] [Created] (SPARK-37451) Performance improvement regressed String to Decimal cast

Raza Jafri created SPARK-37451: -- Summary: Performance improvement regressed String to Decimal cast Key: SPARK-37451 URL: https://issues.apache.org/jira/browse/SPARK-37451 Project: Spark Issue

[jira] [Commented] (SPARK-37367) Reenable exception test in DDLParserSuite.create view -- basic

[ https://issues.apache.org/jira/browse/SPARK-37367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448342#comment-17448342 ] Xiao Li commented on SPARK-37367: - [~beliefer] Can you take a look at this?  > Reenable exception test

[jira] [Updated] (SPARK-37367) Reenable exception test in DDLParserSuite.create view -- basic

[ https://issues.apache.org/jira/browse/SPARK-37367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-37367: Target Version/s: 3.3.0 > Reenable exception test in DDLParserSuite.create view -- basic >

[jira] [Commented] (SPARK-37350) EventLoggingListener keep logging errors after hdfs restart all datanodes

[ https://issues.apache.org/jira/browse/SPARK-37350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448332#comment-17448332 ] Shefron Yudy commented on SPARK-37350: -- [~hyukjin.kwon] ABS EventLogFileWriter with both

[jira] [Commented] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448328#comment-17448328 ] L. C. Hsieh commented on SPARK-37450: - Hmm, this is the optimized plan. {code} == Optimized Logical

[jira] [Comment Edited] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448328#comment-17448328 ] L. C. Hsieh edited comment on SPARK-37450 at 11/24/21, 2:32 AM: Hmm,

[jira] [Commented] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448326#comment-17448326 ] L. C. Hsieh commented on SPARK-37450: - Thanks [~hyukjin.kwon]. I will take a look. > Spark SQL

[jira] [Resolved] (SPARK-37380) Miscellaneous Python lint infra cleanup

[ https://issues.apache.org/jira/browse/SPARK-37380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37380. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34655

[jira] [Updated] (SPARK-37435) Did not find value which can be converted into java.lang.String

[ https://issues.apache.org/jira/browse/SPARK-37435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37435: - Description: Got this following error when loading the saved model. {code} ERROR:ADS Exception

[jira] [Updated] (SPARK-37435) Did not find value which can be converted into java.lang.String

[ https://issues.apache.org/jira/browse/SPARK-37435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37435: - Component/s: ML > Did not find value which can be converted into java.lang.String >

[jira] [Commented] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448321#comment-17448321 ] Hyukjin Kwon commented on SPARK-37449: -- {{applyInPandas}} does not maintain its index. Each pandas

[jira] [Updated] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37449: - Priority: Major (was: Critical) > Side effects between PySpark Pandas UDF and Numpy indexing >

[jira] [Commented] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448320#comment-17448320 ] Hyukjin Kwon commented on SPARK-37450: -- cc [~viirya] FYI > Spark SQL reads unnecessary nested

[jira] [Assigned] (SPARK-37446) hive-2.3.9 related API use invoke method

[ https://issues.apache.org/jira/browse/SPARK-37446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37446: Assignee: Apache Spark > hive-2.3.9 related API use invoke method >

[jira] [Assigned] (SPARK-37446) hive-2.3.9 related API use invoke method

[ https://issues.apache.org/jira/browse/SPARK-37446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37446: Assignee: (was: Apache Spark) > hive-2.3.9 related API use invoke method >

[jira] [Reopened] (SPARK-37446) hive-2.3.9 related API use invoke method

[ https://issues.apache.org/jira/browse/SPARK-37446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-37446: -- > hive-2.3.9 related API use invoke method > > >

[jira] [Assigned] (SPARK-37380) Miscellaneous Python lint infra cleanup

[ https://issues.apache.org/jira/browse/SPARK-37380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37380: Assignee: Nicholas Chammas > Miscellaneous Python lint infra cleanup >

[jira] [Comment Edited] (SPARK-31646) Remove unused registeredConnections counter from ShuffleMetrics

[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448271#comment-17448271 ] Yongjun Zhang edited comment on SPARK-31646 at 11/23/21, 10:47 PM: ---

[jira] [Comment Edited] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448272#comment-17448272 ] Yongjun Zhang edited comment on SPARK-25642 at 11/23/21, 10:46 PM: ---

[jira] [Commented] (SPARK-25642) Add new Metrics in External Shuffle Service to help determine Network performance and Connection Handling capabilities of the Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448272#comment-17448272 ] Yongjun Zhang commented on SPARK-25642: --- Thanks [~pgandhi] .  What I observed is most of the time

[jira] [Commented] (SPARK-31646) Remove unused registeredConnections counter from ShuffleMetrics

[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448271#comment-17448271 ] Yongjun Zhang commented on SPARK-31646: --- Thanks [~mauzhang]  What I observed is most of the time

[jira] [Comment Edited] (SPARK-37432) Driver keep a record of decommission executor

[ https://issues.apache.org/jira/browse/SPARK-37432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448235#comment-17448235 ] Hoa Le edited comment on SPARK-37432 at 11/23/21, 8:43 PM: --- FYI, I had the

[jira] [Commented] (SPARK-37432) Driver keep a record of decommission executor

[ https://issues.apache.org/jira/browse/SPARK-37432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448235#comment-17448235 ] Hoa Le commented on SPARK-37432: FYI, I enabled decommission feature. Thoughts if we can have logic

[jira] [Commented] (SPARK-35885) Use keyserver.ubuntu.com as a keyserver for CRAN

[ https://issues.apache.org/jira/browse/SPARK-35885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448124#comment-17448124 ] Dongjoon Hyun commented on SPARK-35885: --- Thank you for reporting, [~mn-mikke]. > Use

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448101#comment-17448101 ] Michelle m Hovington commented on SPARK-21187: -- Envoyé à partir de

[jira] [Commented] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448001#comment-17448001 ] Jiri Humpolicek commented on SPARK-37450: - same situation with {{size}} function: {code:scala}

[jira] [Updated] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: Let's create a simple Pandas Dataframe with a single column named 'id' with a

[jira] [Updated] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: Let's create a simple Pandas Dataframe with a single column named 'id' that

[jira] [Updated] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: Let's create a simple Pandas Dataframe with a single column named 'id' with a

[jira] [Updated] (SPARK-37449) Side effects between PySpark and Numpy

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: I'm using pygeos 0.11.1. Let's create a simple Pandas Dataframe with a single

[jira] [Updated] (SPARK-37449) Side effects between PySpark Pandas UDF and Numpy indexing

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Summary: Side effects between PySpark Pandas UDF and Numpy indexing (was: Side effects

[jira] [Updated] (SPARK-37449) Side effects between PySpark and Numpy

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: I'm using pygeos 0.11.1. Let's create a simple Pandas Dataframe with a single

[jira] [Updated] (SPARK-37449) Side effects between PySpark and Numpy

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: I'm using pygeos 0.11.1. Let's create a simple Pandas Dataframe with a single

[jira] [Updated] (SPARK-37449) Side effects between PySpark and Numpy

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Summary: Side effects between PySpark and Numpy (was: Side effects between PySpark, Numpy

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447980#comment-17447980 ] Peter Toth commented on SPARK-37259: I've opened a PR: [https://github.com/apache/spark/pull/34693]

[jira] [Created] (SPARK-37450) Spark SQL reads unnecessary nested fields (another type of pruning case)

Jiri Humpolicek created SPARK-37450: --- Summary: Spark SQL reads unnecessary nested fields (another type of pruning case) Key: SPARK-37450 URL: https://issues.apache.org/jira/browse/SPARK-37450

[jira] [Updated] (SPARK-37449) Side effects between PySpark, Numpy and Pygeos

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Priority: Critical (was: Major) > Side effects between PySpark, Numpy and Pygeos >

[jira] [Updated] (SPARK-37449) Side effects between PySpark, Numpy and Pygeos

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Labels: applyInPandas (was: ) > Side effects between PySpark, Numpy and Pygeos >

[jira] [Updated] (SPARK-37449) Side effects between PySpark, Numpy and Pygeos

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Labels: NumPy Pandas Pygeos UDF applyInPandas (was: applyInPandas) > Side effects between

[jira] [Updated] (SPARK-37449) Side effects between PySpark, Numpy and Pygeos

[ https://issues.apache.org/jira/browse/SPARK-37449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos Gameiro updated SPARK-37449: --- Description: I'm using pygeos 0.11.1. Let's create a simple Pandas Dataframe with a single

[jira] [Created] (SPARK-37449) Side effects between PySpark, Numpy and Pygeos

Carlos Gameiro created SPARK-37449: -- Summary: Side effects between PySpark, Numpy and Pygeos Key: SPARK-37449 URL: https://issues.apache.org/jira/browse/SPARK-37449 Project: Spark Issue

[jira] [Assigned] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37259: Assignee: Apache Spark > JDBC read is always going to wrap the query in a select

[jira] [Assigned] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37259: Assignee: (was: Apache Spark) > JDBC read is always going to wrap the query in a

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447972#comment-17447972 ] Apache Spark commented on SPARK-37259: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-37258) Add Volcano support in kubernetes-client

[ https://issues.apache.org/jira/browse/SPARK-37258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447967#comment-17447967 ] Yikun Jiang commented on SPARK-37258: - Volcano extension have been merged in kubernetes client,

[jira] [Commented] (SPARK-11792) SizeEstimator cannot provide a good size estimation of UnsafeHashedRelations

[ https://issues.apache.org/jira/browse/SPARK-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447963#comment-17447963 ] Apache Spark commented on SPARK-11792: -- User 'pan3793' has created a pull request for this issue:

[jira] [Commented] (SPARK-11792) SizeEstimator cannot provide a good size estimation of UnsafeHashedRelations

[ https://issues.apache.org/jira/browse/SPARK-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447961#comment-17447961 ] Apache Spark commented on SPARK-11792: -- User 'pan3793' has created a pull request for this issue:

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447865#comment-17447865 ] Hyukjin Kwon commented on SPARK-22674: -- no biggie either way. I already creatd a PR against that

[jira] [Resolved] (SPARK-36231) Support arithmetic operations of Series containing Decimal(np.nan)

[ https://issues.apache.org/jira/browse/SPARK-36231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36231. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34687

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447859#comment-17447859 ] Sergei Lebedev commented on SPARK-22674: Hi [~hyukjin.kwon], very happy to see you working on

[jira] [Commented] (SPARK-37448) Avoid CurrentOrigin.withOrigin calls in tree transformation methods when partial function isn't defined

[ https://issues.apache.org/jira/browse/SPARK-37448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447842#comment-17447842 ] Josh Rosen commented on SPARK-37448: I plan to work on this. > Avoid CurrentOrigin.withOrigin calls

[jira] [Created] (SPARK-37448) Avoid CurrentOrigin.withOrigin calls in tree transformation methods when partial function isn't defined

Josh Rosen created SPARK-37448: -- Summary: Avoid CurrentOrigin.withOrigin calls in tree transformation methods when partial function isn't defined Key: SPARK-37448 URL:

[jira] [Commented] (SPARK-37391) SIGNIFICANT bottleneck introduced by fix for SPARK-32001

[ https://issues.apache.org/jira/browse/SPARK-37391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447829#comment-17447829 ] Gabor Somogyi commented on SPARK-37391: --- [~hyukjin.kwon] thanks for pinging me. I've added my

[jira] [Commented] (SPARK-37447) Cache LogicalPlan.isStreaming() in a lazy val

[ https://issues.apache.org/jira/browse/SPARK-37447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447828#comment-17447828 ] Apache Spark commented on SPARK-37447: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37447) Cache LogicalPlan.isStreaming() in a lazy val

[ https://issues.apache.org/jira/browse/SPARK-37447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-37447: -- Assignee: Josh Rosen > Cache LogicalPlan.isStreaming() in a lazy val >

[jira] [Commented] (SPARK-37446) hive-2.3.9 related API use invoke method

[ https://issues.apache.org/jira/browse/SPARK-37446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447824#comment-17447824 ] Apache Spark commented on SPARK-37446: -- User 'AngersZh' has created a pull request for this

[jira] [Created] (SPARK-37447) Cache LogicalPlan.isStreaming() in a lazy val

Josh Rosen created SPARK-37447: -- Summary: Cache LogicalPlan.isStreaming() in a lazy val Key: SPARK-37447 URL: https://issues.apache.org/jira/browse/SPARK-37447 Project: Spark Issue Type: