[jira] [Commented] (SPARK-31083) .ClassNotFoundException CoarseGrainedClusterMessages$RetrieveDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-31083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054703#comment-17054703 ] L. C. Hsieh commented on SPARK-31083: - Can you tell more about this issue? I think R

[jira] [Commented] (SPARK-31074) Avro serializer should not fail when a nullable Spark field is written to a non-null Avro column

[ https://issues.apache.org/jira/browse/SPARK-31074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054697#comment-17054697 ] L. C. Hsieh commented on SPARK-31074: - Based on the description, is this the same is

[jira] [Commented] (SPARK-30983) Support more than 5 typed column in typed Dataset.select API

[ https://issues.apache.org/jira/browse/SPARK-30983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054692#comment-17054692 ] L. C. Hsieh commented on SPARK-30983: - For option 2, adding more overloading typed s

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054683#comment-17054683 ] Hyukjin Kwon commented on SPARK-31065: -- Oh, nice catch. Yes, {{null}} types inferre

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054676#comment-17054676 ] Nicholas Chammas commented on SPARK-31043: -- It's working for me now (per my com

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054674#comment-17054674 ] Nicholas Chammas commented on SPARK-31065: -- Thanks for looking into it. I have

[jira] [Updated] (SPARK-30960) add back the legacy date/timestamp format support in CSV/JSON parser

[ https://issues.apache.org/jira/browse/SPARK-30960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30960: -- Parent: SPARK-31085 Issue Type: Sub-task (was: Bug) > add back the legacy date/timest

[jira] [Updated] (SPARK-31091) Revert SPARK-24640 "Return `NULL` from `size(NULL)` by default"

[ https://issues.apache.org/jira/browse/SPARK-31091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31091: -- Reporter: Wenchen Fan (was: Dongjoon Hyun) > Revert SPARK-24640 "Return `NULL` from `size(NUL

[jira] [Assigned] (SPARK-31091) Revert SPARK-24640 "Return `NULL` from `size(NULL)` by default"

[ https://issues.apache.org/jira/browse/SPARK-31091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31091: - Assignee: Wenchen Fan > Revert SPARK-24640 "Return `NULL` from `size(NULL)` by default"

[jira] [Updated] (SPARK-31090) Revert SPARK-25457 IntegralDivide returns data type of the operands

[ https://issues.apache.org/jira/browse/SPARK-31090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31090: -- Reporter: Wenchen Fan (was: Dongjoon Hyun) > Revert SPARK-25457 IntegralDivide returns data t

[jira] [Assigned] (SPARK-31090) Revert SPARK-25457 IntegralDivide returns data type of the operands

[ https://issues.apache.org/jira/browse/SPARK-31090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31090: - Assignee: Wenchen Fan > Revert SPARK-25457 IntegralDivide returns data type of the oper

[jira] [Created] (SPARK-31091) Revert SPARK-24640 "Return `NULL` from `size(NULL)` by default"

Dongjoon Hyun created SPARK-31091: - Summary: Revert SPARK-24640 "Return `NULL` from `size(NULL)` by default" Key: SPARK-31091 URL: https://issues.apache.org/jira/browse/SPARK-31091 Project: Spark

[jira] [Created] (SPARK-31090) Revert SPARK-25457 IntegralDivide returns data type of the operands

Dongjoon Hyun created SPARK-31090: - Summary: Revert SPARK-25457 IntegralDivide returns data type of the operands Key: SPARK-31090 URL: https://issues.apache.org/jira/browse/SPARK-31090 Project: Spark

[jira] [Updated] (SPARK-30886) Deprecate two-parameter TRIM/LTRIM/RTRIM functions

[ https://issues.apache.org/jira/browse/SPARK-30886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30886: -- Parent: SPARK-31085 Issue Type: Sub-task (was: Bug) > Deprecate two-parameter TRIM/LT

[jira] [Resolved] (SPARK-31054) Turn on deprecation in Scala REPL/spark-shell by default

[ https://issues.apache.org/jira/browse/SPARK-31054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31054. --- Resolution: Invalid The approach was invalid because it's too intrusive by turning all Java

[jira] [Closed] (SPARK-31054) Turn on deprecation in Scala REPL/spark-shell by default

[ https://issues.apache.org/jira/browse/SPARK-31054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-31054. - > Turn on deprecation in Scala REPL/spark-shell by default > --

[jira] [Reopened] (SPARK-31054) Turn on deprecation in Scala REPL/spark-shell by default

[ https://issues.apache.org/jira/browse/SPARK-31054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-31054: --- > Turn on deprecation in Scala REPL/spark-shell by default > --

[jira] [Updated] (SPARK-31054) Turn on deprecation in Scala REPL/spark-shell by default

[ https://issues.apache.org/jira/browse/SPARK-31054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31054: -- Parent: SPARK-31085 Issue Type: Sub-task (was: Improvement) > Turn on deprecation in

[jira] [Closed] (SPARK-31089) Add back ImageSchema.readImages in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-31089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-31089. - > Add back ImageSchema.readImages in Spark 3.0 > > >

[jira] [Resolved] (SPARK-31089) Add back ImageSchema.readImages in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-31089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31089. --- Resolution: Won't Do > Add back ImageSchema.readImages in Spark 3.0 > --

[jira] [Created] (SPARK-31089) Add back ImageSchema.readImages in Spark 3.0

Dongjoon Hyun created SPARK-31089: - Summary: Add back ImageSchema.readImages in Spark 3.0 Key: SPARK-31089 URL: https://issues.apache.org/jira/browse/SPARK-31089 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-31089) Add back ImageSchema.readImages in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-31089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31089: -- Reporter: Weichen Xu (was: Dongjoon Hyun) > Add back ImageSchema.readImages in Spark 3.0 > --

[jira] [Assigned] (SPARK-31088) Add back HiveContext and createExternalTable

[ https://issues.apache.org/jira/browse/SPARK-31088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31088: - Assignee: Xiao Li > Add back HiveContext and createExternalTable >

[jira] [Updated] (SPARK-31088) Add back HiveContext and createExternalTable

[ https://issues.apache.org/jira/browse/SPARK-31088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31088: -- Reporter: Xiao Li (was: Dongjoon Hyun) > Add back HiveContext and createExternalTable > -

[jira] [Created] (SPARK-31088) Add back HiveContext and createExternalTable

Dongjoon Hyun created SPARK-31088: - Summary: Add back HiveContext and createExternalTable Key: SPARK-31088 URL: https://issues.apache.org/jira/browse/SPARK-31088 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-31087) Add Back Multiple Removed APIs

[ https://issues.apache.org/jira/browse/SPARK-31087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31087: -- Reporter: Xiao Li (was: Dongjoon Hyun) > Add Back Multiple Removed APIs > ---

[jira] [Created] (SPARK-31087) Add Back Multiple Removed APIs

Dongjoon Hyun created SPARK-31087: - Summary: Add Back Multiple Removed APIs Key: SPARK-31087 URL: https://issues.apache.org/jira/browse/SPARK-31087 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-31087) Add Back Multiple Removed APIs

[ https://issues.apache.org/jira/browse/SPARK-31087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31087: - Assignee: Xiao Li > Add Back Multiple Removed APIs > -- > >

[jira] [Updated] (SPARK-31086) Add Back the Deprecated SQLContext methods

[ https://issues.apache.org/jira/browse/SPARK-31086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31086: -- Reporter: Xiao Li (was: Dongjoon Hyun) > Add Back the Deprecated SQLContext methods > ---

[jira] [Assigned] (SPARK-31086) Add Back the Deprecated SQLContext methods

[ https://issues.apache.org/jira/browse/SPARK-31086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31086: - Assignee: Xiao Li > Add Back the Deprecated SQLContext methods > --

[jira] [Created] (SPARK-31086) Add Back the Deprecated SQLContext methods

Dongjoon Hyun created SPARK-31086: - Summary: Add Back the Deprecated SQLContext methods Key: SPARK-31086 URL: https://issues.apache.org/jira/browse/SPARK-31086 Project: Spark Issue Type: Sub-

[jira] [Updated] (SPARK-31085) Amend Spark's Semantic Versioning Policy

[ https://issues.apache.org/jira/browse/SPARK-31085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31085: -- Issue Type: Umbrella (was: Bug) > Amend Spark's Semantic Versioning Policy >

[jira] [Assigned] (SPARK-31085) Amend Spark's Semantic Versioning Policy

[ https://issues.apache.org/jira/browse/SPARK-31085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31085: - Assignee: Michael Armbrust > Amend Spark's Semantic Versioning Policy > ---

[jira] [Updated] (SPARK-31085) Amend Spark's Semantic Versioning Policy

[ https://issues.apache.org/jira/browse/SPARK-31085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31085: -- Description: This issue tracks all the activity for the following discussion and vote. - http

[jira] [Updated] (SPARK-31085) Amend Spark's Semantic Versioning Policy

[ https://issues.apache.org/jira/browse/SPARK-31085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31085: -- Reporter: Michael Armbrust (was: Dongjoon Hyun) > Amend Spark's Semantic Versioning Policy >

[jira] [Updated] (SPARK-31085) Amend Spark's Semantic Versioning Policy

[ https://issues.apache.org/jira/browse/SPARK-31085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31085: -- Issue Type: Bug (was: Improvement) > Amend Spark's Semantic Versioning Policy > -

[jira] [Created] (SPARK-31085) Amend Spark's Semantic Versioning Policy

Dongjoon Hyun created SPARK-31085: - Summary: Amend Spark's Semantic Versioning Policy Key: SPARK-31085 URL: https://issues.apache.org/jira/browse/SPARK-31085 Project: Spark Issue Type: Improv

[jira] [Comment Edited] (SPARK-20427) Issue with Spark interpreting Oracle datatype NUMBER

[ https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054482#comment-17054482 ] Sunayan Saikia edited comment on SPARK-20427 at 3/9/20, 3:58 AM: -

[jira] [Updated] (SPARK-31084) spark on k8s Exception "Database xxx not found" when hive MetaStoreClient lost connection

[ https://issues.apache.org/jira/browse/SPARK-31084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxiuyuan updated SPARK-31084: --- Description: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke:184 | MetaStoreClient

[jira] [Created] (SPARK-31084) spark on k8s Exception "Database xxx not found" when hive MetaStoreClient lost connection

liuxiuyuan created SPARK-31084: -- Summary: spark on k8s Exception "Database xxx not found" when hive MetaStoreClient lost connection Key: SPARK-31084 URL: https://issues.apache.org/jira/browse/SPARK-31084

[jira] [Created] (SPARK-31083) .ClassNotFoundException CoarseGrainedClusterMessages$RetrieveDelegationTokens

jiama created SPARK-31083: - Summary: .ClassNotFoundException CoarseGrainedClusterMessages$RetrieveDelegationTokens Key: SPARK-31083 URL: https://issues.apache.org/jira/browse/SPARK-31083 Project: Spark

[jira] [Resolved] (SPARK-31073) Add "shuffle write time" to task metrics summary in StagePage.

[ https://issues.apache.org/jira/browse/SPARK-31073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31073. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27837 [https://

[jira] [Created] (SPARK-31082) MapOutputTrackerMaster.getMapLocation can't handle last mapIndex

wuyi created SPARK-31082: Summary: MapOutputTrackerMaster.getMapLocation can't handle last mapIndex Key: SPARK-31082 URL: https://issues.apache.org/jira/browse/SPARK-31082 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31080) Bugs/missing functions in documents

[ https://issues.apache.org/jira/browse/SPARK-31080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054605#comment-17054605 ] Viet commented on SPARK-31080: -- Thank you. > Bugs/missing functions in documents > ---

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054598#comment-17054598 ] Hyukjin Kwon commented on SPARK-31043: -- [~nchammas], can you show the reproducible

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054596#comment-17054596 ] Hyukjin Kwon commented on SPARK-31065: -- There seems two issues. The first is in the

[jira] [Commented] (SPARK-30536) Sort-merge join operator spilling performance improvements

[ https://issues.apache.org/jira/browse/SPARK-30536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054555#comment-17054555 ] shanyu zhao commented on SPARK-30536: - Uploaded two slides to explain the optimizati

[jira] [Updated] (SPARK-30536) Sort-merge join operator spilling performance improvements

[ https://issues.apache.org/jira/browse/SPARK-30536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-30536: Attachment: spark-30536-explained.pdf > Sort-merge join operator spilling performance improvements

[jira] [Commented] (SPARK-31071) Spark Encoders.bean() should allow marking non-null fields in its Spark schema

[ https://issues.apache.org/jira/browse/SPARK-31071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054521#comment-17054521 ] Kyrill Alyoshin commented on SPARK-31071: - _javax.annotation.Nonnull_ - seems li

[jira] [Commented] (SPARK-31071) Spark Encoders.bean() should allow marking non-null fields in its Spark schema

[ https://issues.apache.org/jira/browse/SPARK-31071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054520#comment-17054520 ] L. C. Hsieh commented on SPARK-31071: - What Nonnull annotation we should use? Seems

[jira] [Commented] (SPARK-31080) Bugs/missing functions in documents

[ https://issues.apache.org/jira/browse/SPARK-31080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054516#comment-17054516 ] L. C. Hsieh commented on SPARK-31080: - Then as this is not a bug, I will close this.

[jira] [Resolved] (SPARK-31080) Bugs/missing functions in documents

[ https://issues.apache.org/jira/browse/SPARK-31080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-31080. - Resolution: Not A Bug > Bugs/missing functions in documents > --

[jira] [Commented] (SPARK-31080) Bugs/missing functions in documents

[ https://issues.apache.org/jira/browse/SPARK-31080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054515#comment-17054515 ] L. C. Hsieh commented on SPARK-31080: - There is an article explaining Pivot: https:

[jira] [Assigned] (SPARK-31053) mark connector API as Evolving

[ https://issues.apache.org/jira/browse/SPARK-31053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31053: - Assignee: Wenchen Fan > mark connector API as Evolving > --

[jira] [Resolved] (SPARK-31053) mark connector API as Evolving

[ https://issues.apache.org/jira/browse/SPARK-31053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31053. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27811 [https://

[jira] [Commented] (SPARK-20427) Issue with Spark interpreting Oracle datatype NUMBER

[ https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054482#comment-17054482 ] Sunayan Saikia commented on SPARK-20427: Seems this fix broke the way we could g

[jira] [Created] (SPARK-31081) Make SQLMetrics more readable from UI

wuyi created SPARK-31081: Summary: Make SQLMetrics more readable from UI Key: SPARK-31081 URL: https://issues.apache.org/jira/browse/SPARK-31081 Project: Spark Issue Type: Improvement Compo

[jira] [Resolved] (SPARK-28993) Document Working of Bucketing

[ https://issues.apache.org/jira/browse/SPARK-28993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-28993. -- Resolution: Invalid > Document Working of Bucketing > - >

[jira] [Commented] (SPARK-28993) Document Working of Bucketing

[ https://issues.apache.org/jira/browse/SPARK-28993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054335#comment-17054335 ] Takeshi Yamamuro commented on SPARK-28993: -- I'll close this cuz this is inactiv

[jira] [Resolved] (SPARK-28994) Document working of Adaptive

[ https://issues.apache.org/jira/browse/SPARK-28994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-28994. -- Resolution: Duplicate > Document working of Adaptive > >

[jira] [Commented] (SPARK-28994) Document working of Adaptive

[ https://issues.apache.org/jira/browse/SPARK-28994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054333#comment-17054333 ] Takeshi Yamamuro commented on SPARK-28994: -- I closed this cuz I think this was