[jira] [Resolved] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34046. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31087

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261042#comment-17261042 ] Dongjoon Hyun commented on SPARK-34039: --- Got it. Thanks for the explanation, [~csun]. > [DSv2]

[jira] [Updated] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34039: -- Priority: Minor (was: Major) > [DSv2] ReplaceTable should invalidate cache >

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261036#comment-17261036 ] Chao Sun commented on SPARK-34039: -- I'm not sure if this is a bug since: 1. the command is only

[jira] [Commented] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261022#comment-17261022 ] Apache Spark commented on SPARK-34046: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34046: Assignee: Apache Spark (was: Gengliang Wang) > Use join hint in test cases for Join >

[jira] [Commented] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261021#comment-17261021 ] Apache Spark commented on SPARK-34046: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34046: Assignee: Gengliang Wang (was: Apache Spark) > Use join hint in test cases for Join >

[jira] [Created] (SPARK-34046) Use join hint in test cases for Join

2021-01-07 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-34046: -- Summary: Use join hint in test cases for Join Key: SPARK-34046 URL: https://issues.apache.org/jira/browse/SPARK-34046 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-34005) Update peak memory metrics for each Executor on task end.

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34005. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31029

[jira] [Updated] (SPARK-34005) Update peak memory metrics for each Executor on task end.

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34005: -- Affects Version/s: (was: 3.1.0) > Update peak memory metrics for each Executor on task

[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260856#comment-17260856 ] Ron Hu edited comment on SPARK-26399 at 1/8/21, 5:18 AM: - The initial

[jira] [Assigned] (SPARK-33938) Optimize Like Any/All by LikeSimplification

2021-01-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33938: --- Assignee: jiaan.geng > Optimize Like Any/All by LikeSimplification >

[jira] [Resolved] (SPARK-33938) Optimize Like Any/All by LikeSimplification

2021-01-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33938. - Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31063

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261007#comment-17261007 ] Dongjoon Hyun commented on SPARK-34039: --- [~csun]. Is this an `Improvement` instead of `Bug`? >

[jira] [Assigned] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34039: - Assignee: Chao Sun > [DSv2] ReplaceTable should invalidate cache >

[jira] [Resolved] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34039. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31081

[jira] [Commented] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260969#comment-17260969 ] Apache Spark commented on SPARK-34045: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34045: Assignee: Apache Spark > OneVsRestModel.transform should not call setter of submodels >

[jira] [Commented] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260968#comment-17260968 ] Apache Spark commented on SPARK-34045: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34045: Assignee: (was: Apache Spark) > OneVsRestModel.transform should not call setter of

[jira] [Updated] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-34045: - Description: featuresCol of submodels maybe changed in transform: {code:java} scala> val df =

[jira] [Created] (SPARK-34045) OneVsRestModel.transform should not call setter of submodels

2021-01-07 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-34045: Summary: OneVsRestModel.transform should not call setter of submodels Key: SPARK-34045 URL: https://issues.apache.org/jira/browse/SPARK-34045 Project: Spark

[jira] [Assigned] (SPARK-33818) Doc `spark.sql.parser.quotedRegexColumnNames`

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33818: - Assignee: angerszhu > Doc `spark.sql.parser.quotedRegexColumnNames` >

[jira] [Resolved] (SPARK-33818) Doc `spark.sql.parser.quotedRegexColumnNames`

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33818. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30816

[jira] [Updated] (SPARK-33818) Doc `spark.sql.parser.quotedRegexColumnNames`

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33818: -- Fix Version/s: (was: 3.1.0) 3.1.1 > Doc

[jira] [Updated] (SPARK-33100) Support parse the sql statements with c-style comments

2021-01-07 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33100: - Fix Version/s: 3.0.2 > Support parse the sql statements with c-style comments >

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Commented] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260911#comment-17260911 ] Nasir Ali commented on SPARK-33863: --- [~hyukjin.kwon] and [~viirya] I have simplified example code,

[jira] [Updated] (SPARK-33863) Pyspark UDF wrongly changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: I have a dataframe with a ts (timestamp) column in UTC. If I create a new

[jira] [Updated] (SPARK-33863) Pyspark UDF changes timestamps to UTC

2021-01-07 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-33863: -- Description: *Problem*: If I create a new column using udf, pyspark udf changes timestamps into UTC

[jira] [Commented] (SPARK-33106) Fix sbt resolvers clash

2021-01-07 Thread Alexander Bessonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260903#comment-17260903 ] Alexander Bessonov commented on SPARK-33106: That doesn't seem to fix the issue:

[jira] [Updated] (SPARK-34018) NPE in ExecutorPodsSnapshot

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34018: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Bug) > NPE in ExecutorPodsSnapshot >

[jira] [Resolved] (SPARK-34018) NPE in ExecutorPodsSnapshot

2021-01-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34018. --- Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 31071

[jira] [Updated] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34044: - Fix Version/s: (was: 3.1.0) 3.1.1 > Add

[jira] [Resolved] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34044. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 31085

[jira] [Assigned] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34044: Assignee: Dongjoon Hyun > Add spark.sql.hive.metastore.jars.path to

[jira] [Assigned] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-34041: Assignee: Hyukjin Kwon > Miscellaneous cleanup for new PySpark documentation >

[jira] [Resolved] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-34041. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 31082

[jira] [Assigned] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34044: Assignee: (was: Apache Spark) > Add spark.sql.hive.metastore.jars.path to

[jira] [Assigned] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34044: Assignee: Apache Spark > Add spark.sql.hive.metastore.jars.path to

[jira] [Commented] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260866#comment-17260866 ] Apache Spark commented on SPARK-34044: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260867#comment-17260867 ] Apache Spark commented on SPARK-34044: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-34044) Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md

2021-01-07 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-34044: - Summary: Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md Key: SPARK-34044 URL: https://issues.apache.org/jira/browse/SPARK-34044 Project:

[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26399: --- Description: Add the peak values for the metrics to the stages REST API. Also add a new executorSummary

[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260856#comment-17260856 ] Ron Hu commented on SPARK-26399: The initial description of this jira has this statement:  "filtering

[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26399: --- Attachment: lispark230_restapi_ex2_stages_failedTasks.json > Add new stage-level REST APIs and parameters >

[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260854#comment-17260854 ] Ron Hu commented on SPARK-26399: [^lispark230_restapi_ex2_stages_failedTasks.json] > Add new

[jira] [Commented] (SPARK-33741) Add minimum threshold speculation config

2021-01-07 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260852#comment-17260852 ] Thomas Graves commented on SPARK-33741: --- probably dup of 

[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805 ] Ron Hu edited comment on SPARK-26399 at 1/7/21, 10:23 PM: -- Dr. Elephant

[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805 ] Ron Hu edited comment on SPARK-26399 at 1/7/21, 8:41 PM: - Dr. Elephant

[jira] [Comment Edited] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805 ] Ron Hu edited comment on SPARK-26399 at 1/7/21, 8:40 PM: - Dr. Elephant

[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Hu updated SPARK-26399: --- Attachment: lispark230_restapi_ex2_stages_withSummaries.json > Add new stage-level REST APIs and parameters

[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260808#comment-17260808 ] Ron Hu commented on SPARK-26399: [^lispark230_restapi_ex2_stages_withSummaries.json] > Add new

[jira] [Commented] (SPARK-26399) Add new stage-level REST APIs and parameters

2021-01-07 Thread Ron Hu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260805#comment-17260805 ] Ron Hu commented on SPARK-26399: Dr. Elephant (https://github.com/linkedin/dr-elephant) is a downstream

[jira] [Commented] (SPARK-33661) Unable to load RandomForestClassificationModel trained in Spark 2.x

2021-01-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260789#comment-17260789 ] Sean R. Owen commented on SPARK-33661: -- BTW I think this was actually fixed in

[jira] [Reopened] (SPARK-33661) Unable to load RandomForestClassificationModel trained in Spark 2.x

2021-01-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reopened SPARK-33661: -- > Unable to load RandomForestClassificationModel trained in Spark 2.x >

[jira] [Resolved] (SPARK-33661) Unable to load RandomForestClassificationModel trained in Spark 2.x

2021-01-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-33661. -- Resolution: Duplicate > Unable to load RandomForestClassificationModel trained in Spark 2.x >

[jira] [Created] (SPARK-34043) Linear Discriminant Analysis for dimensionality reduction

2021-01-07 Thread Jose Llorens (Jira)
Jose Llorens created SPARK-34043: Summary: Linear Discriminant Analysis for dimensionality reduction Key: SPARK-34043 URL: https://issues.apache.org/jira/browse/SPARK-34043 Project: Spark

[jira] [Updated] (SPARK-33995) Make datetime addition easier for years, weeks, hours, minutes, and seconds

2021-01-07 Thread Matthew Powers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Powers updated SPARK-33995: --- Affects Version/s: (was: 3.1.0) 3.2.0 > Make datetime addition

[jira] [Commented] (SPARK-33933) Broadcast timeout happened unexpectedly in AQE

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260562#comment-17260562 ] Apache Spark commented on SPARK-33933: -- User 'zhongyu09' has created a pull request for this issue:

[jira] [Commented] (SPARK-32165) SessionState leaks SparkListener with multiple SparkSession

2021-01-07 Thread Vinoo Ganesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260558#comment-17260558 ] Vinoo Ganesh commented on SPARK-32165: -- [~tgraves] - I just added in a description from the comment

[jira] [Updated] (SPARK-32165) SessionState leaks SparkListener with multiple SparkSession

2021-01-07 Thread Vinoo Ganesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-32165: - Description: Copied from [https://github.com/apache/spark/pull/28128#issuecomment-653102770]

[jira] [Updated] (SPARK-32165) SessionState leaks SparkListener with multiple SparkSession

2021-01-07 Thread Vinoo Ganesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-32165: - Description: Copied from [https://github.com/apache/spark/pull/28128#issuecomment-653102770]  

[jira] [Assigned] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34026: Assignee: Apache Spark > DataSource V2: Inject repartition and sort nodes to satisfy

[jira] [Commented] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260399#comment-17260399 ] Apache Spark commented on SPARK-34026: -- User 'aokolnychyi' has created a pull request for this

[jira] [Commented] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260398#comment-17260398 ] Apache Spark commented on SPARK-34026: -- User 'aokolnychyi' has created a pull request for this

[jira] [Assigned] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34026: Assignee: (was: Apache Spark) > DataSource V2: Inject repartition and sort nodes to

[jira] [Resolved] (SPARK-33981) SparkUI: Storage page is empty even if cached

2021-01-07 Thread Maple (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maple resolved SPARK-33981. --- Resolution: Fixed > SparkUI: Storage page is empty even if cached >

[jira] [Updated] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34041: - Description: 1. Add a link of quick start in PySpark docs into "Programming Guides" in Spark

[jira] [Updated] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-07 Thread Marius Butan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius Butan updated SPARK-34042: - Description: In PERMISSIVE mode Given a csv with multiple columns per row, if your file schema

[jira] [Updated] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-07 Thread Marius Butan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius Butan updated SPARK-34042: - Description: In PERMISSIVE mode Given a csv with multiple columns per row, if your file schema

[jira] [Updated] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-07 Thread Marius Butan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius Butan updated SPARK-34042: - Description: In PERMISSIVE mode Given a csv with multiple columns per row, if your file schema

[jira] [Assigned] (SPARK-33933) Broadcast timeout happened unexpectedly in AQE

2021-01-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33933: --- Assignee: Yu Zhong > Broadcast timeout happened unexpectedly in AQE >

[jira] [Resolved] (SPARK-33933) Broadcast timeout happened unexpectedly in AQE

2021-01-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33933. - Fix Version/s: 3.1.1 Resolution: Fixed Issue resolved by pull request 30998

[jira] [Commented] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260327#comment-17260327 ] Apache Spark commented on SPARK-34041: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34041: Assignee: Apache Spark > Miscellaneous cleanup for new PySpark documentation >

[jira] [Commented] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260325#comment-17260325 ] Apache Spark commented on SPARK-34041: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34041: Assignee: (was: Apache Spark) > Miscellaneous cleanup for new PySpark documentation

[jira] [Updated] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-07 Thread Marius Butan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marius Butan updated SPARK-34042: - Description: In PERMISSIVE mode Given a csv with multiple columns, if you have in schema a

[jira] [Created] (SPARK-34042) Column pruning is not working as expected for PERMISIVE mode

2021-01-07 Thread Marius Butan (Jira)
Marius Butan created SPARK-34042: Summary: Column pruning is not working as expected for PERMISIVE mode Key: SPARK-34042 URL: https://issues.apache.org/jira/browse/SPARK-34042 Project: Spark

[jira] [Updated] (SPARK-34041) Miscellaneous cleanup for new PySpark documentation

2021-01-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34041: - Description: 1. Add a link of quick start in PySpark docs into "Programming Guides" in Spark

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260301#comment-17260301 ] Apache Spark commented on SPARK-34039: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34039: Assignee: (was: Apache Spark) > [DSv2] ReplaceTable should invalidate cache >

[jira] [Commented] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260299#comment-17260299 ] Apache Spark commented on SPARK-34039: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34039) [DSv2] ReplaceTable should invalidate cache

2021-01-07 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34039: Assignee: Apache Spark > [DSv2] ReplaceTable should invalidate cache >