[jira] [Commented] (SPARK-28845) Enable spark.sql.execution.sortBeforeRepartition only for retried stages

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035107#comment-17035107 ] Wenchen Fan commented on SPARK-28845: - I'm a little hesitant to abandon the sort approach

[jira] [Resolved] (SPARK-30795) Spark SQL codegen's code() interpolator should treat escapes like Scala's StringContext.s()

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30795. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in

[jira] [Commented] (SPARK-25929) Support metrics with tags

2020-02-11 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035074#comment-17035074 ] John Zhuge commented on SPARK-25929: Yeah, I can feel the pain. When I ingest into InfluxDB, I have

[jira] [Updated] (SPARK-30796) Add parameter position for REGEXP_REPLACE

2020-02-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30796: --- Parent: SPARK-27764 Issue Type: Sub-task (was: New Feature) > Add parameter position for

[jira] [Commented] (SPARK-30796) Add parameter position for REGEXP_REPLACE

2020-02-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035066#comment-17035066 ] jiaan.geng commented on SPARK-30796: I'm working on. > Add parameter position for REGEXP_REPLACE >

[jira] [Created] (SPARK-30796) Add parameter position for REGEXP_REPLACE

2020-02-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30796: -- Summary: Add parameter position for REGEXP_REPLACE Key: SPARK-30796 URL: https://issues.apache.org/jira/browse/SPARK-30796 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-30722) Document type hints in pandas UDF

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30722: Assignee: Hyukjin Kwon > Document type hints in pandas UDF >

[jira] [Resolved] (SPARK-30722) Document type hints in pandas UDF

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30722. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27466

[jira] [Resolved] (SPARK-30780) LocalRelation should use emptyRDD if it is empty

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30780. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27530

[jira] [Created] (SPARK-30795) Spark SQL codegen's code() interpolator should treat escapes like Scala's StringContext.s()

2020-02-11 Thread Kris Mok (Jira)
Kris Mok created SPARK-30795: Summary: Spark SQL codegen's code() interpolator should treat escapes like Scala's StringContext.s() Key: SPARK-30795 URL: https://issues.apache.org/jira/browse/SPARK-30795

[jira] [Created] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

2020-02-11 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-30794: - Summary: Stage Level scheduling: Add ability to set off heap memory Key: SPARK-30794 URL: https://issues.apache.org/jira/browse/SPARK-30794 Project: Spark

[jira] [Commented] (SPARK-27913) Spark SQL's native ORC reader implements its own schema evolution

2020-02-11 Thread Giri (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034875#comment-17034875 ] Giri commented on SPARK-27913: -- This issue doesn't exist in  *spark spark-3.0.0-preview2 and also in spark

[jira] [Commented] (SPARK-28845) Enable spark.sql.execution.sortBeforeRepartition only for retried stages

2020-02-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034755#comment-17034755 ] Thomas Graves commented on SPARK-28845: --- [~cloud_fan] [~XuanYuan] I wanted to followup on this

[jira] [Created] (SPARK-30793) Wrong truncations of timestamps before the epoch to minutes and seconds

2020-02-11 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30793: -- Summary: Wrong truncations of timestamps before the epoch to minutes and seconds Key: SPARK-30793 URL: https://issues.apache.org/jira/browse/SPARK-30793 Project: Spark

[jira] [Created] (SPARK-30792) Dataframe .limit() performance improvements

2020-02-11 Thread Nathan Grand (Jira)
Nathan Grand created SPARK-30792: Summary: Dataframe .limit() performance improvements Key: SPARK-30792 URL: https://issues.apache.org/jira/browse/SPARK-30792 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30783) Hive 2.3 profile should exclude hive-service-rpc

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30783. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27533

[jira] [Assigned] (SPARK-27545) Update the Documentation for CACHE TABLE and UNCACHE TABLE

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27545: --- Assignee: Rakesh Raushan (was: hantiantian) > Update the Documentation for CACHE TABLE

[jira] [Resolved] (SPARK-30754) Reuse results of floorDiv in calculations of floorMod in DateTimeUtils

2020-02-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30754. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27491

[jira] [Assigned] (SPARK-30754) Reuse results of floorDiv in calculations of floorMod in DateTimeUtils

2020-02-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30754: Assignee: Maxim Gekk > Reuse results of floorDiv in calculations of floorMod in

[jira] [Comment Edited] (SPARK-27710) ClassNotFoundException: $line196400984558.$read$ in OuterScopes

2020-02-11 Thread Jelmer Kuperus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034532#comment-17034532 ] Jelmer Kuperus edited comment on SPARK-27710 at 2/11/20 3:07 PM: - This

[jira] [Commented] (SPARK-27710) ClassNotFoundException: $line196400984558.$read$ in OuterScopes

2020-02-11 Thread Jelmer Kuperus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034532#comment-17034532 ] Jelmer Kuperus commented on SPARK-27710: This also happens in Apache Toree   {code:java} val

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-02-11 Thread Jorge Machado (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034492#comment-17034492 ] Jorge Machado commented on SPARK-24615: --- Yeah, that was my question. Thanks for the response. I

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-02-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034490#comment-17034490 ] Thomas Graves commented on SPARK-24615: --- This is purely a scheduling feature and Spark will assign

[jira] [Commented] (SPARK-27545) Update the Documentation for CACHE TABLE and UNCACHE TABLE

2020-02-11 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034479#comment-17034479 ] Rakesh Raushan commented on SPARK-27545: Please assign this to me. Thanks > Update the

[jira] [Updated] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-30791: --- Description: Sometimes, we want to check whether two dataframes are the same. There is already an

[jira] [Commented] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034475#comment-17034475 ] Weichen Xu commented on SPARK-30791: [~liangz] will work on this. :) > Dataframe add sameResult and

[jira] [Assigned] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-11 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-30791: -- Assignee: Liang Zhang > Dataframe add sameResult and sementicHash method >

[jira] [Created] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-11 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-30791: -- Summary: Dataframe add sameResult and sementicHash method Key: SPARK-30791 URL: https://issues.apache.org/jira/browse/SPARK-30791 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-30790) The datatype of map() should be map

2020-02-11 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034461#comment-17034461 ] Rakesh Raushan commented on SPARK-30790: Should i expose a legacy configuration for mapType as

[jira] [Created] (SPARK-30790) The datatype of map() should be map

2020-02-11 Thread Rakesh Raushan (Jira)
Rakesh Raushan created SPARK-30790: -- Summary: The datatype of map() should be map Key: SPARK-30790 URL: https://issues.apache.org/jira/browse/SPARK-30790 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27545) Update the Documentation for CACHE TABLE and UNCACHE TABLE

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27545: Summary: Update the Documentation for CACHE TABLE and UNCACHE TABLE (was: Uncache table needs to

[jira] [Updated] (SPARK-27545) Update the Documentation for CACHE TABLE and UNCACHE TABLE

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27545: Issue Type: Documentation (was: Bug) > Update the Documentation for CACHE TABLE and UNCACHE

[jira] [Updated] (SPARK-27545) Update the Documentation for CACHE TABLE and UNCACHE TABLE

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27545: Description: spark-sql> cache table v1 as select * from a; spark-sql> uncache table v1;

[jira] [Resolved] (SPARK-27545) Uncache table needs to delete the temporary view created when the cache table is executed.

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27545. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27090

[jira] [Assigned] (SPARK-27545) Uncache table needs to delete the temporary view created when the cache table is executed.

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27545: --- Assignee: hantiantian > Uncache table needs to delete the temporary view created when the

[jira] [Assigned] (SPARK-30326) Raise exception if analyzer exceed max iterations

2020-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30326: --- Assignee: Xin Wu > Raise exception if analyzer exceed max iterations >

[jira] [Created] (SPARK-30789) Support IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE

2020-02-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30789: -- Summary: Support IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE Key: SPARK-30789 URL: https://issues.apache.org/jira/browse/SPARK-30789 Project:

[jira] [Commented] (SPARK-30789) Support IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE

2020-02-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034347#comment-17034347 ] jiaan.geng commented on SPARK-30789: I will working on. > Support IGNORE | RESPECT) NULLS for

[jira] [Updated] (SPARK-30786) Block replication is not retried on other BlockManagers when it fails on 1 of the peers

2020-02-11 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-30786: - Component/s: Spark Core > Block replication is not retried on other BlockManagers when it fails

[jira] [Updated] (SPARK-30787) Add Generic Algorithm optimizer feature to spark-ml

2020-02-11 Thread louischoi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] louischoi updated SPARK-30787: -- Target Version/s: (was: 2.4.5) > Add Generic Algorithm optimizer feature to spark-ml >

[jira] [Created] (SPARK-30788) Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters

2020-02-11 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-30788: -- Summary: Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters Key: SPARK-30788 URL: https://issues.apache.org/jira/browse/SPARK-30788

[jira] [Created] (SPARK-30787) Add Generic Algorithm optimizer feature to spark-ml

2020-02-11 Thread louischoi (Jira)
louischoi created SPARK-30787: - Summary: Add Generic Algorithm optimizer feature to spark-ml Key: SPARK-30787 URL: https://issues.apache.org/jira/browse/SPARK-30787 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2020-02-11 Thread Jorge Machado (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034277#comment-17034277 ] Jorge Machado commented on SPARK-24615: --- [~tgraves] thanks for the input. It would be great to

[jira] [Commented] (SPARK-29474) CLI support for Spark-on-Docker-on-Yarn

2020-02-11 Thread Abhijeet Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034244#comment-17034244 ] Abhijeet Singh commented on SPARK-29474: Thanks for this feature suggestion [~adam.antal]. Is

[jira] [Resolved] (SPARK-29462) The data type of "array()" should be array

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29462. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27521

[jira] [Assigned] (SPARK-29462) The data type of "array()" should be array

2020-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29462: Assignee: Hyukjin Kwon > The data type of "array()" should be array >