[jira] [Commented] (SPARK-35210) Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331160#comment-17331160 ] Apache Spark commented on SPARK-35210: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-35210) Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331159#comment-17331159 ] Apache Spark commented on SPARK-35210: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-35210) Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35210: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade Jetty to 9.4.40 to fix ERR_CONNEC

[jira] [Assigned] (SPARK-35210) Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35210: Assignee: Apache Spark (was: Kousuke Saruta) > Upgrade Jetty to 9.4.40 to fix ERR_CONNEC

[jira] [Created] (SPARK-35210) Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue

2021-04-23 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-35210: -- Summary: Upgrade Jetty to 9.4.40 to fix ERR_CONNECTION_RESET issue Key: SPARK-35210 URL: https://issues.apache.org/jira/browse/SPARK-35210 Project: Spark

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331157#comment-17331157 ] Dongjoon Hyun commented on SPARK-35196: --- Ya, Sorry for the negative opinion. There

[jira] [Commented] (SPARK-35199) Tasks are failing with zstd default of spark.shuffle.mapStatus.compression.codec

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331155#comment-17331155 ] Dongjoon Hyun commented on SPARK-35199: --- According to error logs, it seems that yo

[jira] [Commented] (SPARK-35199) Tasks are failing with zstd default of spark.shuffle.mapStatus.compression.codec

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331154#comment-17331154 ] Dongjoon Hyun commented on SPARK-35199: --- Well, I'd recommend to use ZSTD at Apach

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331153#comment-17331153 ] Hyukjin Kwon commented on SPARK-35196: -- I see. Thanks Dongjoon for clarification!

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331152#comment-17331152 ] Dongjoon Hyun commented on SPARK-35196: --- In addition, even with Hadoop 3.1, the of

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331150#comment-17331150 ] Dongjoon Hyun commented on SPARK-35196: --- Hi, [~lausen] and [~hyukjin.kwon]. We sti

[jira] [Commented] (SPARK-33195) stages/stage UI page fails to load when spark reverse proxy is enabled

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331144#comment-17331144 ] Apache Spark commented on SPARK-33195: -- User 'mdianjun' has created a pull request

[jira] [Assigned] (SPARK-33195) stages/stage UI page fails to load when spark reverse proxy is enabled

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33195: Assignee: (was: Apache Spark) > stages/stage UI page fails to load when spark reverse

[jira] [Assigned] (SPARK-33195) stages/stage UI page fails to load when spark reverse proxy is enabled

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33195: Assignee: Apache Spark > stages/stage UI page fails to load when spark reverse proxy is e

[jira] [Commented] (SPARK-33195) stages/stage UI page fails to load when spark reverse proxy is enabled

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331143#comment-17331143 ] Apache Spark commented on SPARK-33195: -- User 'mdianjun' has created a pull request

[jira] [Commented] (SPARK-35168) mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

2021-04-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331132#comment-17331132 ] Kent Yao commented on SPARK-35168: -- Thanks [~dongjoon] > mapred.reduce.tasks should be

[jira] [Created] (SPARK-35209) CLONE - CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Jira
Kadir Selçuk created SPARK-35209: Summary: CLONE - CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes Key: SPARK-35209 URL: https://issues.apache.org/jira/browse/SPARK-35

[jira] [Commented] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331117#comment-17331117 ] Kadir Selçuk commented on SPARK-35204: -- Sorunları çözmek  > CatalystTypeConverters

[jira] [Commented] (SPARK-35160) Spark application submitted despite failing to get Hive delegation token

2021-04-23 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331114#comment-17331114 ] Manu Zhang commented on SPARK-35160: [~hyukjin.kwon], Thanks for reminder. I've add

[jira] [Updated] (SPARK-35160) Spark application submitted despite failing to get Hive delegation token

2021-04-23 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated SPARK-35160: --- Description: Currently, when running on YARN and failing to get Hive delegation token, a Spark SQL

[jira] [Commented] (SPARK-28247) Flaky test: "query without test harness" in ContinuousSuite

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331103#comment-17331103 ] Apache Spark commented on SPARK-28247: -- User 'zsxwing' has created a pull request f

[jira] [Created] (SPARK-35208) Add docs for LATERAL subqueries

2021-04-23 Thread Allison Wang (Jira)
Allison Wang created SPARK-35208: Summary: Add docs for LATERAL subqueries Key: SPARK-35208 URL: https://issues.apache.org/jira/browse/SPARK-35208 Project: Spark Issue Type: Task Co

[jira] [Commented] (SPARK-35133) EXPLAIN CODEGEN does not work with AQE

2021-04-23 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331096#comment-17331096 ] Wei Xue commented on SPARK-35133: - Totally understand. But turning off AQE is sometimes

[jira] [Updated] (SPARK-35207) hash() and other hash builtins do not normalize negative zero

2021-04-23 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-35207: -- Description: I would generally expect that {{x = y => hash(x) = hash(y)}}. However +-0 hash t

[jira] [Updated] (SPARK-35207) hash() and other hash builtins do not normalize negative zero

2021-04-23 Thread Tim Armstrong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated SPARK-35207: -- Description: I would generally expect that {{x = y => hash( x ) = hash( y )}}. However +-0 ha

[jira] [Created] (SPARK-35207) hash() and other hash builtins do not normalize negative zero

2021-04-23 Thread Tim Armstrong (Jira)
Tim Armstrong created SPARK-35207: - Summary: hash() and other hash builtins do not normalize negative zero Key: SPARK-35207 URL: https://issues.apache.org/jira/browse/SPARK-35207 Project: Spark

[jira] [Commented] (SPARK-35168) mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331059#comment-17331059 ] Dongjoon Hyun commented on SPARK-35168: --- Thank you, [~Qin Yao]. I converted this i

[jira] [Updated] (SPARK-35168) mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum

2021-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35168: -- Parent: SPARK-33828 Issue Type: Sub-task (was: Bug) > mapred.reduce.tasks should be s

[jira] [Resolved] (SPARK-34297) Add metrics for data loss and offset out range for KafkaMicroBatchStream

2021-04-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34297. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31398 [https://gith

[jira] [Commented] (SPARK-35133) EXPLAIN CODEGEN does not work with AQE

2021-04-23 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331011#comment-17331011 ] Cheng Su commented on SPARK-35133: -- btw just to provide more context, I am running into

[jira] [Commented] (SPARK-35156) Thrown java.lang.NoClassDefFoundError when using spark-submit

2021-04-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331003#comment-17331003 ] L. C. Hsieh commented on SPARK-35156: - > Can you just use a later version? it may be

[jira] [Commented] (SPARK-35156) Thrown java.lang.NoClassDefFoundError when using spark-submit

2021-04-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330999#comment-17330999 ] L. C. Hsieh commented on SPARK-35156: - > Do you mean Branch-3.0 is affected alone?

[jira] [Updated] (SPARK-35156) Thrown java.lang.NoClassDefFoundError when using spark-submit

2021-04-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-35156: Description: Got NoClassDefFoundError when run spark-submit to submit Spark app to K8S cluster.

[jira] [Commented] (SPARK-35133) EXPLAIN CODEGEN does not work with AQE

2021-04-23 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330982#comment-17330982 ] Cheng Su commented on SPARK-35133: -- When ever developers/users want to debug generated

[jira] [Updated] (SPARK-35013) Spark allows to set spark.driver.cores=0

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-35013: - Issue Type: Improvement (was: Bug) > Spark allows to set spark.driver.cores=0 > ---

[jira] [Commented] (SPARK-35013) Spark allows to set spark.driver.cores=0

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330978#comment-17330978 ] Sean R. Owen commented on SPARK-35013: -- I can't think of a reason to allow 0 cores.

[jira] [Commented] (SPARK-35027) Close the inputStream in FileAppender when writing the logs failure

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330976#comment-17330976 ] Sean R. Owen commented on SPARK-35027: -- Are you sure? stop() is called on an error

[jira] [Resolved] (SPARK-35046) Wrong memory allocation on standalone mode cluster

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-35046. -- Resolution: Invalid > Wrong memory allocation on standalone mode cluster > ---

[jira] [Resolved] (SPARK-35054) Getting Critical Vulnerability CVE-2021-20231 on spark 3.0.0 branch

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-35054. -- Resolution: Invalid There's no info about what these are or if they even affect Spark. > Gett

[jira] [Commented] (SPARK-35156) Thrown java.lang.NoClassDefFoundError when using spark-submit

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330971#comment-17330971 ] Sean R. Owen commented on SPARK-35156: -- Can you just use a later version? it may be

[jira] [Resolved] (SPARK-35193) Scala/Java compatibility issue Re: how to use externalResource in java transformer from Scala Transformer?

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-35193. -- Resolution: Invalid I think this should be a question to the user@ list - I don't see reason t

[jira] [Resolved] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available

2021-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34430. -- Fix Version/s: (was: 3.0.0) Target Version/s: (was: 3.0.0) Resolution: W

[jira] [Updated] (SPARK-35197) Accumulators Explore Page on Spark UI on History Server

2021-04-23 Thread Frida Montserrat Pulido Padilla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frida Montserrat Pulido Padilla updated SPARK-35197: Summary: Accumulators Explore Page on Spark UI on History S

[jira] [Commented] (SPARK-35133) EXPLAIN CODEGEN does not work with AQE

2021-04-23 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330920#comment-17330920 ] Wei Xue commented on SPARK-35133: - I'm not against fixing it. But just wondering is it e

[jira] [Comment Edited] (SPARK-34458) Spark-hive: apache hive dependency with CVEs

2021-04-23 Thread Bhupesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330873#comment-17330873 ] Bhupesh edited comment on SPARK-34458 at 4/23/21, 4:48 PM: --- I

[jira] [Assigned] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-35204: Assignee: Wenchen Fan > CatalystTypeConverters of date/timestamp should accept both the old and n

[jira] [Resolved] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-35204. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32312 [https://github.com

[jira] [Comment Edited] (SPARK-34458) Spark-hive: apache hive dependency with CVEs

2021-04-23 Thread Bhupesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330873#comment-17330873 ] Bhupesh edited comment on SPARK-34458 at 4/23/21, 3:44 PM: --- I

[jira] [Commented] (SPARK-34458) Spark-hive: apache hive dependency with CVEs

2021-04-23 Thread Bhupesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330873#comment-17330873 ] Bhupesh commented on SPARK-34458: - I found that, this is already upgraded twice. Followi

[jira] [Updated] (SPARK-35162) New SQL functions: TRY_ADD/TRY_DIVIDE

2021-04-23 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-35162: --- Summary: New SQL functions: TRY_ADD/TRY_DIVIDE (was: New SQL functions: TRY_ADD/TRY_SUBTRAC

[jira] [Updated] (SPARK-35161) Error-handling SQL functions

2021-04-23 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-35161: --- Summary: Error-handling SQL functions (was: Safe version SQL functions) > Error-handling SQ

[jira] [Updated] (SPARK-35161) Error-handling SQL functions

2021-04-23 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-35161: --- Description: Create new Error-handling version SQL functions for existing SQL functions/ope

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330803#comment-17330803 ] Hyukjin Kwon commented on SPARK-35196: -- Yeah, I think it's all implemented properly

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Leonard Lausen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330795#comment-17330795 ] Leonard Lausen commented on SPARK-35196: Great. Adding the alias should be stra

[jira] [Commented] (SPARK-35206) Extract common get project path ability as function to SparkFunctionSuite

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330460#comment-17330460 ] Apache Spark commented on SPARK-35206: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-35206) Extract common get project path ability as function to SparkFunctionSuite

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35206: Assignee: Apache Spark > Extract common get project path ability as function to SparkFunc

[jira] [Commented] (SPARK-35206) Extract common get project path ability as function to SparkFunctionSuite

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330461#comment-17330461 ] Apache Spark commented on SPARK-35206: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-35206) Extract common get project path ability as function to SparkFunctionSuite

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35206: Assignee: (was: Apache Spark) > Extract common get project path ability as function t

[jira] [Created] (SPARK-35206) Extract common get project path ability as function to SparkFunctionSuite

2021-04-23 Thread wuyi (Jira)
wuyi created SPARK-35206: Summary: Extract common get project path ability as function to SparkFunctionSuite Key: SPARK-35206 URL: https://issues.apache.org/jira/browse/SPARK-35206 Project: Spark Is

[jira] [Resolved] (SPARK-35201) Format empty grouping set exception in CUBE/ROLLUP

2021-04-23 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-35201. -- Fix Version/s: 3.2.0 Assignee: angerszhu Resolution: Fixed Resolved by

[jira] [Resolved] (SPARK-35123) read partitioned parquet: my_col=NOW replaced by on read()

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35123. -- Resolution: Duplicate > read partitioned parquet: my_col=NOW replaced by on read() >

[jira] [Commented] (SPARK-35123) read partitioned parquet: my_col=NOW replaced by on read()

2021-04-23 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330391#comment-17330391 ] Max Gekk commented on SPARK-35123: -- The PR [https://github.com/apache/spark/pull/31549]

[jira] [Commented] (SPARK-35133) EXPLAIN CODEGEN does not work with AQE

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330389#comment-17330389 ] Hyukjin Kwon commented on SPARK-35133: -- cc [~maryannxue] and [~Ngone51] FYI > EXPL

[jira] [Commented] (SPARK-35123) read partitioned parquet: my_col=NOW replaced by on read()

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330390#comment-17330390 ] Hyukjin Kwon commented on SPARK-35123: -- [~maxgekk] FYI > read partitioned parquet:

[jira] [Commented] (SPARK-35149) I am facing this issue regularly, how to fix this issue.

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330388#comment-17330388 ] Hyukjin Kwon commented on SPARK-35149: -- For questions, please use Spark mailing lis

[jira] [Resolved] (SPARK-35149) I am facing this issue regularly, how to fix this issue.

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35149. -- Resolution: Invalid > I am facing this issue regularly, how to fix this issue. > -

[jira] [Commented] (SPARK-35154) Rpc env not shutdown when shutdown method call by endpoint onStop

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330387#comment-17330387 ] Hyukjin Kwon commented on SPARK-35154: -- {{RpcEndpoint}} isn't an API. > Rpc env no

[jira] [Resolved] (SPARK-35154) Rpc env not shutdown when shutdown method call by endpoint onStop

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35154. -- Resolution: Invalid > Rpc env not shutdown when shutdown method call by endpoint onStop >

[jira] [Commented] (SPARK-35156) Thrown java.lang.NoClassDefFoundError when using spark-submit

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330386#comment-17330386 ] Hyukjin Kwon commented on SPARK-35156: -- [~viirya] no big deal but: {quote} Master,

[jira] [Commented] (SPARK-35160) Spark application submitted despite failing to get Hive delegation token

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330385#comment-17330385 ] Hyukjin Kwon commented on SPARK-35160: -- [~mauzhang] if this is a question, it shoul

[jira] [Commented] (SPARK-35176) Raise TypeError in inappropriate type case rather than ValueError

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330383#comment-17330383 ] Hyukjin Kwon commented on SPARK-35176: -- [~yikunkero] Please go ahead for a PR > R

[jira] [Resolved] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35184. -- Resolution: Cannot Reproduce > Filtering a dataframe after groupBy and user-define-aggregate-f

[jira] [Commented] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330381#comment-17330381 ] Hyukjin Kwon commented on SPARK-35184: -- Seems like it works in the latest master br

[jira] [Assigned] (SPARK-35169) Wrong result of min ANSI interval division by -1

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35169: Assignee: Apache Spark > Wrong result of min ANSI interval division by -1 > -

[jira] [Assigned] (SPARK-35169) Wrong result of min ANSI interval division by -1

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35169: Assignee: (was: Apache Spark) > Wrong result of min ANSI interval division by -1 > --

[jira] [Resolved] (SPARK-35190) all columns are read even if column pruning applies when spark3.0 read table written by spark2.2

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35190. -- Resolution: Duplicate > all columns are read even if column pruning applies when spark3.0 read

[jira] [Resolved] (SPARK-35191) all columns are read even if column pruning applies when spark3.0 read table written by spark2.2

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35191. -- Resolution: Duplicate > all columns are read even if column pruning applies when spark3.0 read

[jira] [Commented] (SPARK-35196) DataFrameWriter.text support zstd compression

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330379#comment-17330379 ] Hyukjin Kwon commented on SPARK-35196: -- It does support, and you can specify {{org

[jira] [Commented] (SPARK-35199) Tasks are failing with zstd default of spark.shuffle.mapStatus.compression.codec

2021-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330377#comment-17330377 ] Hyukjin Kwon commented on SPARK-35199: -- cc [~dongjoon] FYI > Tasks are failing wit

[jira] [Commented] (SPARK-35169) Wrong result of min ANSI interval division by -1

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330375#comment-17330375 ] Apache Spark commented on SPARK-35169: -- User 'AngersZh' has created a pull requ

[jira] [Commented] (SPARK-35169) Wrong result of min ANSI interval division by -1

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330374#comment-17330374 ] Apache Spark commented on SPARK-35169: -- User 'AngersZh' has created a pull requ

[jira] [Updated] (SPARK-35205) Simplify org.apache.hive.service.cli.OperationType.getOperationType by using a hashMap.

2021-04-23 Thread akiyamaneko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] akiyamaneko updated SPARK-35205: Summary: Simplify org.apache.hive.service.cli.OperationType.getOperationType by using a hashMap.

[jira] [Updated] (SPARK-35205) Refractor org.apache.hive.service.cli.OperationType.getOperationType by using a hashMap.

2021-04-23 Thread akiyamaneko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] akiyamaneko updated SPARK-35205: Summary: Refractor org.apache.hive.service.cli.OperationType.getOperationType by using a hashMap.

[jira] [Assigned] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35205: Assignee: (was: Apache Spark) > Simplify operationType.getOperationType by using a ha

[jira] [Updated] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread akiyamaneko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] akiyamaneko updated SPARK-35205: Description: SImply the *getOperationType* method in `org.apache.hive.service.cli.OperationType`.

[jira] [Assigned] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35205: Assignee: (was: Apache Spark) > Simplify operationType.getOperationType by using a ha

[jira] [Commented] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330371#comment-17330371 ] Apache Spark commented on SPARK-35205: -- User 'kyoty' has created a pull request for

[jira] [Assigned] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35205: Assignee: Apache Spark > Simplify operationType.getOperationType by using a hashMap. > --

[jira] [Created] (SPARK-35205) Simplify operationType.getOperationType by using a hashMap.

2021-04-23 Thread akiyamaneko (Jira)
akiyamaneko created SPARK-35205: --- Summary: Simplify operationType.getOperationType by using a hashMap. Key: SPARK-35205 URL: https://issues.apache.org/jira/browse/SPARK-35205 Project: Spark Is

[jira] [Assigned] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35204: Assignee: Apache Spark > CatalystTypeConverters of date/timestamp should accept both the

[jira] [Assigned] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35204: Assignee: (was: Apache Spark) > CatalystTypeConverters of date/timestamp should accep

[jira] [Commented] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330332#comment-17330332 ] Apache Spark commented on SPARK-35204: -- User 'cloud-fan' has created a pull request

[jira] [Comment Edited] (SPARK-35176) Raise TypeError in inappropriate type case rather than ValueError

2021-04-23 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17329971#comment-17329971 ] Yikun Jiang edited comment on SPARK-35176 at 4/23/21, 9:14 AM: ---

[jira] [Commented] (SPARK-35169) Wrong result of min ANSI interval division by -1

2021-04-23 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330241#comment-17330241 ] angerszhu commented on SPARK-35169: --- It's guava IntMath.divude's bug.. > Wrong result

[jira] [Created] (SPARK-35204) CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes

2021-04-23 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-35204: --- Summary: CatalystTypeConverters of date/timestamp should accept both the old and new Java time classes Key: SPARK-35204 URL: https://issues.apache.org/jira/browse/SPARK-35204

[jira] [Commented] (SPARK-35088) Accept ANSI intervals by the Sequence expression

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330232#comment-17330232 ] Apache Spark commented on SPARK-35088: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-35088) Accept ANSI intervals by the Sequence expression

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35088: Assignee: Apache Spark > Accept ANSI intervals by the Sequence expression > -

[jira] [Assigned] (SPARK-35088) Accept ANSI intervals by the Sequence expression

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35088: Assignee: (was: Apache Spark) > Accept ANSI intervals by the Sequence expression > --

[jira] [Commented] (SPARK-35088) Accept ANSI intervals by the Sequence expression

2021-04-23 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330230#comment-17330230 ] Apache Spark commented on SPARK-35088: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-35078) Migrate to transformWithPruning or resolveWithPruning for expression rules

2021-04-23 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-35078: -- Assignee: Yingyi Bu > Migrate to transformWithPruning or resolveWithPruning for expre

  1   2   >