[jira] [Commented] (SPARK-32189) Development - Setting up PyCharm

[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197402#comment-17197402 ] Apache Spark commented on SPARK-32189: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32189) Development - Setting up PyCharm

[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32189: Assignee: Apache Spark (was: Haejoon Lee) > Development - Setting up PyCharm >

[jira] [Assigned] (SPARK-32189) Development - Setting up PyCharm

[ https://issues.apache.org/jira/browse/SPARK-32189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32189: Assignee: Haejoon Lee (was: Apache Spark) > Development - Setting up PyCharm >

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197395#comment-17197395 ] Lauri Koobas commented on SPARK-29900: -- The problem actually arose from using the

[jira] [Commented] (SPARK-32186) Development - Debugging

[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197393#comment-17197393 ] Apache Spark commented on SPARK-32186: -- User 'itholic' has created a pull request for this issue:

[jira] [Commented] (SPARK-32186) Development - Debugging

[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197391#comment-17197391 ] Apache Spark commented on SPARK-32186: -- User 'itholic' has created a pull request for this issue:

[jira] [Resolved] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

[ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32903. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29776

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197377#comment-17197377 ] Gengliang Wang commented on SPARK-27589: [~dongjoon] Thanks for reminder. I will revisit this

[jira] [Commented] (SPARK-32906) Struct field names should not change after normalizing floats

[ https://issues.apache.org/jira/browse/SPARK-32906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197374#comment-17197374 ] Apache Spark commented on SPARK-32906: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32906) Struct field names should not change after normalizing floats

[ https://issues.apache.org/jira/browse/SPARK-32906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32906: Assignee: Apache Spark > Struct field names should not change after normalizing floats >

[jira] [Commented] (SPARK-32906) Struct field names should not change after normalizing floats

[ https://issues.apache.org/jira/browse/SPARK-32906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197373#comment-17197373 ] Apache Spark commented on SPARK-32906: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32906) Struct field names should not change after normalizing floats

[ https://issues.apache.org/jira/browse/SPARK-32906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32906: Assignee: (was: Apache Spark) > Struct field names should not change after

[jira] [Commented] (SPARK-28396) Add PathCatalog for data source V2

[ https://issues.apache.org/jira/browse/SPARK-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197370#comment-17197370 ] Gengliang Wang commented on SPARK-28396: [~dongjoon] It's very likely that this will be done.

[jira] [Created] (SPARK-32906) Struct field names should not change after normalizing floats

Takeshi Yamamuro created SPARK-32906: Summary: Struct field names should not change after normalizing floats Key: SPARK-32906 URL: https://issues.apache.org/jira/browse/SPARK-32906 Project: Spark

[jira] [Commented] (SPARK-32180) Getting Started - Installation

[ https://issues.apache.org/jira/browse/SPARK-32180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197358#comment-17197358 ] Apache Spark commented on SPARK-32180: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-32180) Getting Started - Installation

[ https://issues.apache.org/jira/browse/SPARK-32180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197357#comment-17197357 ] Apache Spark commented on SPARK-32180: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197341#comment-17197341 ] wuyi commented on SPARK-32898: -- I think the issue is(for executorRunTimeMs): Before a task reaches to

[jira] [Commented] (SPARK-18409) LSH approxNearestNeighbors should use approxQuantile instead of sort

[ https://issues.apache.org/jira/browse/SPARK-18409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197338#comment-17197338 ] Apache Spark commented on SPARK-18409: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-32905) ApplicationMaster fails to receive UpdateDelegationTokens message

[ https://issues.apache.org/jira/browse/SPARK-32905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32905: Assignee: (was: Apache Spark) > ApplicationMaster fails to receive

[jira] [Commented] (SPARK-32905) ApplicationMaster fails to receive UpdateDelegationTokens message

[ https://issues.apache.org/jira/browse/SPARK-32905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197333#comment-17197333 ] Apache Spark commented on SPARK-32905: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32905) ApplicationMaster fails to receive UpdateDelegationTokens message

[ https://issues.apache.org/jira/browse/SPARK-32905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32905: Assignee: Apache Spark > ApplicationMaster fails to receive UpdateDelegationTokens

[jira] [Created] (SPARK-32905) ApplicationMaster fails to receive UpdateDelegationTokens message

Kent Yao created SPARK-32905: Summary: ApplicationMaster fails to receive UpdateDelegationTokens message Key: SPARK-32905 URL: https://issues.apache.org/jira/browse/SPARK-32905 Project: Spark

[jira] [Assigned] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

[ https://issues.apache.org/jira/browse/SPARK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-26425: Assignee: Jungtaek Lim (was: Tathagata Das) > Add more constraint checks in file

[jira] [Resolved] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

[ https://issues.apache.org/jira/browse/SPARK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-26425. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 25965

[jira] [Created] (SPARK-32904) pyspark.mllib.evaluation.MulticlassMetrics needs to swap the results of precision( ) and recall( )

TinaLi created SPARK-32904: -- Summary: pyspark.mllib.evaluation.MulticlassMetrics needs to swap the results of precision( ) and recall( ) Key: SPARK-32904 URL: https://issues.apache.org/jira/browse/SPARK-32904

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197250#comment-17197250 ] Terry Kim commented on SPARK-29900: --- When you run `show tables`, you get the `isTemporary` column, so

[jira] [Commented] (SPARK-30283) V2 Command logical plan should use UnresolvedV2Relation for a table

[ https://issues.apache.org/jira/browse/SPARK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197249#comment-17197249 ] Terry Kim commented on SPARK-30283: --- [~cloud_fan], this work stopped when alter table hit a bump in

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197228#comment-17197228 ] Dongjoon Hyun commented on SPARK-27589: --- [~Gengliang.Wang]. What is the new JIRA issue for

[jira] [Commented] (SPARK-28396) Add PathCatalog for data source V2

[ https://issues.apache.org/jira/browse/SPARK-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197225#comment-17197225 ] Dongjoon Hyun commented on SPARK-28396: --- Hi, [~Gengliang.Wang] This is closed as 'Won't Fix'. So,

[jira] [Commented] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

[ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197223#comment-17197223 ] Apache Spark commented on SPARK-32903: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

[ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32903: Assignee: Apache Spark (was: L. C. Hsieh) > GeneratePredicate should be able to

[jira] [Assigned] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

[ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32903: Assignee: L. C. Hsieh (was: Apache Spark) > GeneratePredicate should be able to

[jira] [Created] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

L. C. Hsieh created SPARK-32903: --- Summary: GeneratePredicate should be able to eliminate common sub-expressions Key: SPARK-32903 URL: https://issues.apache.org/jira/browse/SPARK-32903 Project: Spark

[jira] [Assigned] (SPARK-32295) Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

[ https://issues.apache.org/jira/browse/SPARK-32295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32295: Assignee: (was: Apache Spark) > Add not null and size > 0 filters before inner

[jira] [Assigned] (SPARK-32295) Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

[ https://issues.apache.org/jira/browse/SPARK-32295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32295: Assignee: Apache Spark > Add not null and size > 0 filters before inner explode to

[jira] [Reopened] (SPARK-32295) Add not null and size > 0 filters before inner explode to benefit from predicate pushdown

[ https://issues.apache.org/jira/browse/SPARK-32295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis reopened SPARK-32295: > Add not null and size > 0 filters before inner explode to benefit from > predicate pushdown >

[jira] [Commented] (SPARK-24994) Add UnwrapCastInBinaryComparison optimizer to simplify literal types

[ https://issues.apache.org/jira/browse/SPARK-24994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197198#comment-17197198 ] Apache Spark commented on SPARK-24994: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32890) Pass all `sql/hive` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-32890: Assignee: Yang Jie > Pass all `sql/hive` module UTs in Scala 2.13 >

[jira] [Resolved] (SPARK-32890) Pass all `sql/hive` module UTs in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-32890. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29760

[jira] [Commented] (SPARK-27589) Spark file source V2

[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197143#comment-17197143 ] Thomas Graves commented on SPARK-27589: --- somewhat related, I was looking through the v2 code for

[jira] [Updated] (SPARK-32897) SparkSession.builder.getOrCreate should not show deprecation warning of SQLContext

[ https://issues.apache.org/jira/browse/SPARK-32897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-32897: -- Affects Version/s: (was: 2.4.7) > SparkSession.builder.getOrCreate should not show

[jira] [Resolved] (SPARK-32897) SparkSession.builder.getOrCreate should not show deprecation warning of SQLContext

[ https://issues.apache.org/jira/browse/SPARK-32897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-32897. --- Fix Version/s: 3.1.0 3.0.2 Assignee: Hyukjin Kwon

[jira] [Assigned] (SPARK-32816) Planner error when aggregating multiple distinct DECIMAL columns

[ https://issues.apache.org/jira/browse/SPARK-32816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32816: --- Assignee: Linhong Liu > Planner error when aggregating multiple distinct DECIMAL columns >

[jira] [Resolved] (SPARK-32816) Planner error when aggregating multiple distinct DECIMAL columns

[ https://issues.apache.org/jira/browse/SPARK-32816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32816. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29673

[jira] [Closed] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Punit Shah closed SPARK-32888. -- Resolved by adding documentation > reading a parallized rdd with two identical records results in a zero

[jira] [Commented] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197073#comment-17197073 ] L. C. Hsieh commented on SPARK-32888: - Yes, there is difference. But it is due to reading file and

[jira] [Commented] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197069#comment-17197069 ] Punit Shah commented on SPARK-32888: Thank you for your reply [~viirya]  However what I've noticed

[jira] [Updated] (SPARK-32898) totalExecutorRunTimeMs is too big

[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-32898: - Description: This might be because of incorrectly calculating executorRunTimeMs in Executor.scala The

[jira] [Commented] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197046#comment-17197046 ] L. C. Hsieh commented on SPARK-32888: - Reading csv files is simple. We can just remove first line.

[jira] [Assigned] (SPARK-32850) Simply the RPC message flow of decommission

[ https://issues.apache.org/jira/browse/SPARK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32850: --- Assignee: wuyi > Simply the RPC message flow of decommission >

[jira] [Resolved] (SPARK-32850) Simply the RPC message flow of decommission

[ https://issues.apache.org/jira/browse/SPARK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32850. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29722

[jira] [Comment Edited] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196985#comment-17196985 ] Punit Shah edited comment on SPARK-32888 at 9/16/20, 2:55 PM: -- Why do we

[jira] [Resolved] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

[ https://issues.apache.org/jira/browse/SPARK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32706. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29731

[jira] [Assigned] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

[ https://issues.apache.org/jira/browse/SPARK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32706: --- Assignee: Yuming Wang > Poor performance when casting invalid decimal string to decimal

[jira] [Commented] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196985#comment-17196985 ] Punit Shah commented on SPARK-32888: Why do we remove lines that are the same as the header? The

[jira] [Assigned] (SPARK-32902) Logging plan changes for AQE

[ https://issues.apache.org/jira/browse/SPARK-32902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32902: Assignee: Apache Spark > Logging plan changes for AQE > > >

[jira] [Commented] (SPARK-32902) Logging plan changes for AQE

[ https://issues.apache.org/jira/browse/SPARK-32902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196977#comment-17196977 ] Apache Spark commented on SPARK-32902: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32902) Logging plan changes for AQE

[ https://issues.apache.org/jira/browse/SPARK-32902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32902: Assignee: (was: Apache Spark) > Logging plan changes for AQE >

[jira] [Commented] (SPARK-32902) Logging plan changes for AQE

[ https://issues.apache.org/jira/browse/SPARK-32902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196978#comment-17196978 ] Apache Spark commented on SPARK-32902: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-32287) Flaky Test: ExecutorAllocationManagerSuite.add executors default profile

[ https://issues.apache.org/jira/browse/SPARK-32287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196976#comment-17196976 ] Apache Spark commented on SPARK-32287: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-32287) Flaky Test: ExecutorAllocationManagerSuite.add executors default profile

[ https://issues.apache.org/jira/browse/SPARK-32287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196975#comment-17196975 ] Apache Spark commented on SPARK-32287: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Created] (SPARK-32902) Logging plan changes for AQE

Takeshi Yamamuro created SPARK-32902: Summary: Logging plan changes for AQE Key: SPARK-32902 URL: https://issues.apache.org/jira/browse/SPARK-32902 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196972#comment-17196972 ] Thomas Graves commented on SPARK-32898: --- [~linhongliu-db] can you please provide more of a

[jira] [Updated] (SPARK-32894) Timestamp cast in exernal orc table

[ https://issues.apache.org/jira/browse/SPARK-32894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32894: -- Summary: Timestamp cast in exernal orc table (was: Timestamp cast in exernal ocr table) >

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Labels: correct (was: ) > When pyspark.sql.functions.lit() function is used with dataframe

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Labels: correctness (was: correct) > When pyspark.sql.functions.lit() function is used with

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32635: -- Priority: Blocker (was: Major) > When pyspark.sql.functions.lit() function is used with

[jira] [Assigned] (SPARK-32185) User Guide - Monitoring

[ https://issues.apache.org/jira/browse/SPARK-32185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32185: Assignee: Abhijeet Prasad > User Guide - Monitoring > --- > >

[jira] [Commented] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196931#comment-17196931 ] Apache Spark commented on SPARK-32900: -- User 'tomvanbussel' has created a pull request for this

[jira] [Assigned] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32900: Assignee: (was: Apache Spark) > UnsafeExternalSorter.SpillableIterator cannot spill

[jira] [Assigned] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32900: Assignee: Apache Spark > UnsafeExternalSorter.SpillableIterator cannot spill when there

[jira] [Commented] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196910#comment-17196910 ] Apache Spark commented on SPARK-32635: -- User 'peter-toth' has created a pull request for this

[jira] [Assigned] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32635: Assignee: Apache Spark > When pyspark.sql.functions.lit() function is used with

[jira] [Assigned] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32635: Assignee: (was: Apache Spark) > When pyspark.sql.functions.lit() function is used

[jira] [Assigned] (SPARK-32814) Metaclasses are broken for a few classes in Python 3

[ https://issues.apache.org/jira/browse/SPARK-32814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32814: Assignee: Maciej Szymkiewicz > Metaclasses are broken for a few classes in Python 3 >

[jira] [Resolved] (SPARK-32814) Metaclasses are broken for a few classes in Python 3

[ https://issues.apache.org/jira/browse/SPARK-32814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32814. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29664

[jira] [Assigned] (SPARK-32835) Add withField to PySpark Column class

[ https://issues.apache.org/jira/browse/SPARK-32835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32835: Assignee: Adam Binford > Add withField to PySpark Column class >

[jira] [Resolved] (SPARK-32835) Add withField to PySpark Column class

[ https://issues.apache.org/jira/browse/SPARK-32835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32835. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29699

[jira] [Assigned] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32888: Assignee: L. C. Hsieh > reading a parallized rdd with two identical records results in a

[jira] [Resolved] (SPARK-32888) reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv

[ https://issues.apache.org/jira/browse/SPARK-32888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32888. -- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution:

[jira] [Created] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

Tom van Bussel created SPARK-32901: -- Summary: UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling Key: SPARK-32901 URL: https://issues.apache.org/jira/browse/SPARK-32901

[jira] [Updated] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom van Bussel updated SPARK-32900: --- Description: In order to determine whether {{UnsafeExternalSorter.SpillableIterator}} has

[jira] [Created] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

Tom van Bussel created SPARK-32900: -- Summary: UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used. Key: SPARK-32900 URL:

[jira] [Comment Edited] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196812#comment-17196812 ] Lauri Koobas edited comment on SPARK-29900 at 9/16/20, 9:18 AM: Bringing

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196812#comment-17196812 ] Lauri Koobas commented on SPARK-29900: -- Bringing up a related point - `show tables in ` always

[jira] [Comment Edited] (SPARK-32894) Timestamp cast in exernal ocr table

[ https://issues.apache.org/jira/browse/SPARK-32894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196783#comment-17196783 ] Grigory Skvortsov edited comment on SPARK-32894 at 9/16/20, 8:32 AM: -

[jira] [Commented] (SPARK-32894) Timestamp cast in exernal ocr table

[ https://issues.apache.org/jira/browse/SPARK-32894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196783#comment-17196783 ] Grigory Skvortsov commented on SPARK-32894: --- >From hiveCli using following code: > Timestamp

[jira] [Assigned] (SPARK-32899) Support submit application with user-defined cluster manager

[ https://issues.apache.org/jira/browse/SPARK-32899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32899: Assignee: (was: Apache Spark) > Support submit application with user-defined cluster

[jira] [Commented] (SPARK-32899) Support submit application with user-defined cluster manager

[ https://issues.apache.org/jira/browse/SPARK-32899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196754#comment-17196754 ] Apache Spark commented on SPARK-32899: -- User 'ConeyLiu' has created a pull request for this issue:

[jira] [Commented] (SPARK-32899) Support submit application with user-defined cluster manager

[ https://issues.apache.org/jira/browse/SPARK-32899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196753#comment-17196753 ] Apache Spark commented on SPARK-32899: -- User 'ConeyLiu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32899) Support submit application with user-defined cluster manager

[ https://issues.apache.org/jira/browse/SPARK-32899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32899: Assignee: Apache Spark > Support submit application with user-defined cluster manager >

[jira] [Created] (SPARK-32899) Support submit application with user-defined cluster manager

Xianyang Liu created SPARK-32899: Summary: Support submit application with user-defined cluster manager Key: SPARK-32899 URL: https://issues.apache.org/jira/browse/SPARK-32899 Project: Spark

[jira] [Comment Edited] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196721#comment-17196721 ] Aman Rastogi edited comment on SPARK-32778 at 9/16/20, 6:46 AM: I have

[jira] [Reopened] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Rastogi reopened SPARK-32778: -- I have reproduced the issue with v2.4.4. Code is also similar as it was in v2.2.0 

[jira] [Updated] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Rastogi updated SPARK-32778: - Affects Version/s: (was: 2.2.0) 2.4.4 > Accidental Data Deletion on