[jira] [Updated] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Pedamallu updated SPARK-34107: --- Attachment: SHS_Profiling_Sorted.csv > Spark History not loading when service has to

[jira] [Commented] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264672#comment-17264672 ] Shashank Pedamallu commented on SPARK-34107: Screenshot of the spark history

[jira] [Updated] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Pedamallu updated SPARK-34107: --- Attachment: blank_shs.png > Spark History not loading when service has to load 300k

[jira] [Assigned] (SPARK-34096) Improve performance for nth_value ignore nulls over offset window

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34096: Assignee: Apache Spark > Improve performance for nth_value ignore nulls over offset windo

[jira] [Assigned] (SPARK-34096) Improve performance for nth_value ignore nulls over offset window

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34096: Assignee: (was: Apache Spark) > Improve performance for nth_value ignore nulls over o

[jira] [Commented] (SPARK-34096) Improve performance for nth_value ignore nulls over offset window

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264649#comment-17264649 ] Apache Spark commented on SPARK-34096: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264639#comment-17264639 ] Apache Spark commented on SPARK-34110: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34110: Assignee: (was: Apache Spark) > Upgrade ZooKeeper to 3.6.2 >

[jira] [Assigned] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34110: Assignee: Apache Spark > Upgrade ZooKeeper to 3.6.2 > -- > >

[jira] [Updated] (SPARK-34096) Improve performance for nth_value ignore nulls over offset window

2021-01-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-34096: --- Summary: Improve performance for nth_value ignore nulls over offset window (was: Improve performanc

[jira] [Updated] (SPARK-34096) Improve performance for nth_value ignore nulls with offset window

2021-01-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-34096: --- Summary: Improve performance for nth_value ignore nulls with offset window (was: Improve performanc

[jira] [Created] (SPARK-34112) Upgrade ORC

2021-01-13 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-34112: - Summary: Upgrade ORC Key: SPARK-34112 URL: https://issues.apache.org/jira/browse/SPARK-34112 Project: Spark Issue Type: Sub-task Components: Buil

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264631#comment-17264631 ] Hyukjin Kwon commented on SPARK-34111: -- I just marked it as a blocker because the d

[jira] [Commented] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264630#comment-17264630 ] Dongjoon Hyun commented on SPARK-34110: --- Thank you, [~yumwang]! > Upgrade ZooKeep

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264629#comment-17264629 ] Kent Yao commented on SPARK-34111: -- thanks [~hyukjin.kwon] for pinging me.  > Deconfli

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264628#comment-17264628 ] Dongjoon Hyun commented on SPARK-34111: --- Oh, is this a blocker? > Deconflict the

[jira] [Commented] (SPARK-23431) Expose the new executor memory metrics at the stage level

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264626#comment-17264626 ] Dongjoon Hyun commented on SPARK-23431: --- Thank you! > Expose the new executor mem

[jira] [Commented] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264621#comment-17264621 ] Yuming Wang commented on SPARK-34110: - Another issue is: {noformat} 21/01/13 22:49:

[jira] [Commented] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264620#comment-17264620 ] Hyukjin Kwon commented on SPARK-34111: -- cc [~Qin Yao] FYI > Deconflict the jars ja

[jira] [Updated] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34111: - Description: After SPARK-33705, we now happened to have two jars in the release artifact with H

[jira] [Created] (SPARK-34111) Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar

2021-01-13 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-34111: Summary: Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar Key: SPARK-34111 URL: https://issues.apache.org/jira/browse/SPARK-34111 Pro

[jira] [Updated] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34110: Description: When running Spark on JDK 14: {noformat} 21/01/13 20:25:32,533 WARN [Driver-SendThre

[jira] [Resolved] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-34106. -- Resolution: Duplicate > Hide FValueTest and AnovaTest > - > >

[jira] [Created] (SPARK-34110) Upgrade ZooKeeper to 3.6.2

2021-01-13 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-34110: --- Summary: Upgrade ZooKeeper to 3.6.2 Key: SPARK-34110 URL: https://issues.apache.org/jira/browse/SPARK-34110 Project: Spark Issue Type: Sub-task Compo

[jira] [Comment Edited] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264608#comment-17264608 ] Chao Sun edited comment on SPARK-33507 at 1/14/21, 5:23 AM:

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264608#comment-17264608 ] Chao Sun commented on SPARK-33507: -- Thanks [~hyukjin.kwon]. From my side, there is no r

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264601#comment-17264601 ] Hyukjin Kwon commented on SPARK-33507: -- Hey guys, I think we should just start RC r

[jira] [Assigned] (SPARK-34081) Only pushdown LeftSemi/LeftAnti over Aggregate if join can be planned as broadcast join

2021-01-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34081: --- Assignee: Yuming Wang > Only pushdown LeftSemi/LeftAnti over Aggregate if join can be plann

[jira] [Resolved] (SPARK-34081) Only pushdown LeftSemi/LeftAnti over Aggregate if join can be planned as broadcast join

2021-01-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34081. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31145 [https://gith

[jira] [Updated] (SPARK-34096) Improve performance for nth_value ignore nulls

2021-01-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-34096: --- Description: The current {code:java} UnboundedOffsetWindowFunctionFrame {code} and {code:java} Unbo

[jira] [Updated] (SPARK-34096) Improve performance for nth_value ignore nulls

2021-01-13 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-34096: --- Summary: Improve performance for nth_value ignore nulls (was: Improve performance for nth_value ign

[jira] [Updated] (SPARK-34109) Killing executors excluded on failure, results in additional executors being marked as excluded due to fetch failures

2021-01-13 Thread Aaruna Godthi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaruna Godthi updated SPARK-34109: -- Description: Configuration:   {code:java} spark.excludeOnFailure.enabled: true # aka deprecat

[jira] [Created] (SPARK-34109) Killing executors excluded on failure, results in additional executors being marked as excluded due to fetch failures

2021-01-13 Thread Aaruna Godthi (Jira)
Aaruna Godthi created SPARK-34109: - Summary: Killing executors excluded on failure, results in additional executors being marked as excluded due to fetch failures Key: SPARK-34109 URL: https://issues.apache.org/ji

[jira] [Resolved] (SPARK-34086) RaiseError generates too much code and may fails codegen in length check for char varchar

2021-01-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34086. - Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31168 [https://gith

[jira] [Assigned] (SPARK-34086) RaiseError generates too much code and may fails codegen in length check for char varchar

2021-01-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34086: --- Assignee: Kent Yao > RaiseError generates too much code and may fails codegen in length che

[jira] [Updated] (SPARK-23431) Expose the new executor memory metrics at the stage level

2021-01-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-23431: --- Fix Version/s: 3.1.0 > Expose the new executor memory metrics at the stage level > -

[jira] [Comment Edited] (SPARK-23431) Expose the new executor memory metrics at the stage level

2021-01-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264571#comment-17264571 ] Gengliang Wang edited comment on SPARK-23431 at 1/14/21, 3:20 AM:

[jira] [Commented] (SPARK-23431) Expose the new executor memory metrics at the stage level

2021-01-13 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264571#comment-17264571 ] Gengliang Wang commented on SPARK-23431: [~dongjoon]Done. Sorry for missing the

[jira] [Updated] (SPARK-34108) Caching with permanent view doesn't work in certain cases

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Summary: Caching with permanent view doesn't work in certain cases (was: Caching doesn't work completel

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in certain cases. For instance, in the fo

[jira] [Updated] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-34108: - Description: Currently, caching a permanent view doesn't work in some cases. For instance, in the follo

[jira] [Created] (SPARK-34108) Caching doesn't work completely with permanent view

2021-01-13 Thread Chao Sun (Jira)
Chao Sun created SPARK-34108: Summary: Caching doesn't work completely with permanent view Key: SPARK-34108 URL: https://issues.apache.org/jira/browse/SPARK-34108 Project: Spark Issue Type: Sub-t

[jira] [Assigned] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34106: Assignee: Apache Spark (was: zhengruifeng) > Hide FValueTest and AnovaTest > ---

[jira] [Commented] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264567#comment-17264567 ] Apache Spark commented on SPARK-34106: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34106: Assignee: zhengruifeng (was: Apache Spark) > Hide FValueTest and AnovaTest > ---

[jira] [Updated] (SPARK-33311) Improve semantics for REFRESH TABLE

2021-01-13 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-33311: - Parent: SPARK-33507 Issue Type: Sub-task (was: Improvement) > Improve semantics for REFRESH TAB

[jira] [Updated] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Pedamallu updated SPARK-34107: --- Description: Spark History Service is having trouble loading when loading initially

[jira] [Updated] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashank Pedamallu updated SPARK-34107: --- Description: Spark History Service is having trouble loading when loading initially

[jira] [Created] (SPARK-34107) Spark History not loading when service has to load 300k applications initially from S3

2021-01-13 Thread Shashank Pedamallu (Jira)
Shashank Pedamallu created SPARK-34107: -- Summary: Spark History not loading when service has to load 300k applications initially from S3 Key: SPARK-34107 URL: https://issues.apache.org/jira/browse/SPARK-34107

[jira] [Created] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-34106: Summary: Hide FValueTest and AnovaTest Key: SPARK-34106 URL: https://issues.apache.org/jira/browse/SPARK-34106 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-34106) Hide FValueTest and AnovaTest

2021-01-13 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-34106: Assignee: zhengruifeng > Hide FValueTest and AnovaTest > - >

[jira] [Updated] (SPARK-33557) spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33557: - Fix Version/s: 3.0.2 > spark.storage.blockManagerSlaveTimeoutMs default value does not follow >

[jira] [Commented] (SPARK-34097) overflow for datetime datatype when creating stride + JDBC parallel read

2021-01-13 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264538#comment-17264538 ] Takeshi Yamamuro commented on SPARK-34097: -- Ah, I see. Thanks for the report. T

[jira] [Commented] (SPARK-34097) overflow for datetime datatype when creating stride + JDBC parallel read

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264516#comment-17264516 ] Hyukjin Kwon commented on SPARK-34097: -- cc [~maropu] FYI > overflow for datetime d

[jira] [Resolved] (SPARK-34100) pyspark 2.4 packages can't be installed via pip on Amazon Linux 2

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34100. -- Resolution: Cannot Reproduce > pyspark 2.4 packages can't be installed via pip on Amazon Linux

[jira] [Commented] (SPARK-34100) pyspark 2.4 packages can't be installed via pip on Amazon Linux 2

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264510#comment-17264510 ] Hyukjin Kwon commented on SPARK-34100: -- If this is fixed upstream, it's best to ide

[jira] [Updated] (SPARK-34097) overflow for datetime datatype when creating stride + JDBC parallel read

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34097: - Target Version/s: (was: 3.0.1) > overflow for datetime datatype when creating stride + JDBC p

[jira] [Resolved] (SPARK-34075) Hidden directories are being listed for partition inference

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34075. -- Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31169 [https://gi

[jira] [Assigned] (SPARK-34075) Hidden directories are being listed for partition inference

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34075: Assignee: Gengliang Wang > Hidden directories are being listed for partition inference >

[jira] [Assigned] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34103: Assignee: Dongjoon Hyun > Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x > --

[jira] [Resolved] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34103. -- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-33557) spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264496#comment-17264496 ] Apache Spark commented on SPARK-33557: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-33557) spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264494#comment-17264494 ] Apache Spark commented on SPARK-33557: -- User 'dongjoon-hyun' has created a pull req

[jira] [Resolved] (SPARK-34051) Support 32-bit unicode escape in string literals

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34051. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31096 [https://gi

[jira] [Resolved] (SPARK-34068) Remove redundant collection conversion in Spark code

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34068. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31125 [https://gi

[jira] [Assigned] (SPARK-34068) Remove redundant collection conversion in Spark code

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34068: Assignee: Yang Jie > Remove redundant collection conversion in Spark code > -

[jira] [Created] (SPARK-34105) In addition to killing exlcuded/flakey executors which should support decommissioning

2021-01-13 Thread Holden Karau (Jira)
Holden Karau created SPARK-34105: Summary: In addition to killing exlcuded/flakey executors which should support decommissioning Key: SPARK-34105 URL: https://issues.apache.org/jira/browse/SPARK-34105

[jira] [Created] (SPARK-34104) Allow users to specify a maximum decommissioning time

2021-01-13 Thread Holden Karau (Jira)
Holden Karau created SPARK-34104: Summary: Allow users to specify a maximum decommissioning time Key: SPARK-34104 URL: https://issues.apache.org/jira/browse/SPARK-34104 Project: Spark Issue T

[jira] [Assigned] (SPARK-34104) Allow users to specify a maximum decommissioning time

2021-01-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-34104: Assignee: Holden Karau > Allow users to specify a maximum decommissioning time >

[jira] [Commented] (SPARK-34104) Allow users to specify a maximum decommissioning time

2021-01-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264488#comment-17264488 ] Holden Karau commented on SPARK-34104: -- I'm working on this. > Allow users to spec

[jira] [Commented] (SPARK-23431) Expose the new executor memory metrics at the stage level

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264468#comment-17264468 ] Dongjoon Hyun commented on SPARK-23431: --- Hi, [~Gengliang.Wang]. What is the fixed

[jira] [Issue Comment Deleted] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34103: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue: htt

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264463#comment-17264463 ] Apache Spark commented on SPARK-23429: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264462#comment-17264462 ] Apache Spark commented on SPARK-23429: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264460#comment-17264460 ] Apache Spark commented on SPARK-23429: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264461#comment-17264461 ] Apache Spark commented on SPARK-23429: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34103: -- Affects Version/s: 3.1.0 3.0.0 3.0.1 > Fix MiMaE

[jira] [Commented] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264459#comment-17264459 ] Apache Spark commented on SPARK-34103: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264458#comment-17264458 ] Apache Spark commented on SPARK-34103: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34103: Assignee: Apache Spark > Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x > ---

[jira] [Assigned] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34103: Assignee: (was: Apache Spark) > Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to

[jira] [Created] (SPARK-34103) Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x

2021-01-13 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-34103: - Summary: Fix MiMaExcludes by moving SPARK-23429 from 2.4.x to 3.0.x Key: SPARK-34103 URL: https://issues.apache.org/jira/browse/SPARK-34103 Project: Spark

[jira] [Assigned] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-23429: - Assignee: Edward Lu > Add executor memory metrics to heartbeat and expose in executors

[jira] [Created] (SPARK-34102) Spark SQL cannot escape both \ and other special characters

2021-01-13 Thread Noah Kawasaki (Jira)
Noah Kawasaki created SPARK-34102: - Summary: Spark SQL cannot escape both \ and other special characters Key: SPARK-34102 URL: https://issues.apache.org/jira/browse/SPARK-34102 Project: Spark

[jira] [Commented] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2021-01-13 Thread Noah Kawasaki (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264452#comment-17264452 ] Noah Kawasaki commented on SPARK-17647: --- I can also confirm that this issue is not

[jira] [Commented] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264451#comment-17264451 ] Apache Spark commented on SPARK-34101: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34101: Assignee: Kousuke Saruta (was: Apache Spark) > Make spark-sql CLI configurable for the b

[jira] [Assigned] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34101: Assignee: Apache Spark (was: Kousuke Saruta) > Make spark-sql CLI configurable for the b

[jira] [Commented] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264450#comment-17264450 ] Apache Spark commented on SPARK-34101: -- User 'sarutak' has created a pull request f

[jira] [Updated] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34101: --- Description: Like Hive CLI, spark-sql CLI accepts hive.cli.print.header property and we can

[jira] [Updated] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34101: --- Description: Like Hive CLI, spark-sql CLI accept hive.cli.print.header property and we can

[jira] [Created] (SPARK-34101) Make spark-sql CLI configurable for the behavior of printing header by SET command

2021-01-13 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-34101: -- Summary: Make spark-sql CLI configurable for the behavior of printing header by SET command Key: SPARK-34101 URL: https://issues.apache.org/jira/browse/SPARK-34101

[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2021-01-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264325#comment-17264325 ] Dongjoon Hyun commented on SPARK-19169: --- [~angerszhuuu]. Given the context, this l

[jira] [Comment Edited] (SPARK-34100) pyspark 2.4 packages can't be installed via pip on Amazon Linux 2

2021-01-13 Thread Devin Boyer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264309#comment-17264309 ] Devin Boyer edited comment on SPARK-34100 at 1/13/21, 5:43 PM: ---

[jira] [Commented] (SPARK-34100) pyspark 2.4 packages can't be installed via pip on Amazon Linux 2

2021-01-13 Thread Devin Boyer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264309#comment-17264309 ] Devin Boyer commented on SPARK-34100: - Noting that I found a workaround here: it app

[jira] [Commented] (SPARK-32333) Drop references to Master

2021-01-13 Thread Neil Shah-Quinn (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264303#comment-17264303 ] Neil Shah-Quinn commented on SPARK-32333: - I'm glad there's a plan to improve th

[jira] [Created] (SPARK-34100) pyspark 2.4 packages can't be installed via pip on Amazon Linux 2

2021-01-13 Thread Devin Boyer (Jira)
Devin Boyer created SPARK-34100: --- Summary: pyspark 2.4 packages can't be installed via pip on Amazon Linux 2 Key: SPARK-34100 URL: https://issues.apache.org/jira/browse/SPARK-34100 Project: Spark

[jira] [Assigned] (SPARK-34070) Replaces find and emptiness check with exists.

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-34070: Assignee: Yang Jie > Replaces find and emptiness check with exists. > ---

[jira] [Resolved] (SPARK-34070) Replaces find and emptiness check with exists.

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-34070. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31130 [https://gi

[jira] [Updated] (SPARK-34070) Replaces find and emptiness check with exists.

2021-01-13 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34070: - Priority: Trivial (was: Minor) > Replaces find and emptiness check with exists. > -

  1   2   >