[jira] [Updated] (SPARK-31454) An optimized K-Means based on DenseMatrix and GEMM

[ https://issues.apache.org/jira/browse/SPARK-31454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochang Wu updated SPARK-31454: - Description: The main computations in K-Means are calculating distances between individual poin

[jira] [Updated] (SPARK-31454) An optimized K-Means based on DenseMatrix and GEMM

[ https://issues.apache.org/jira/browse/SPARK-31454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochang Wu updated SPARK-31454: - Affects Version/s: (was: 3.0.0) 3.1.0 Description: The main

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084552#comment-17084552 ] Hongze Zhang commented on SPARK-31437: -- Thanks [~tgraves]. I got your point of maki

[jira] [Updated] (SPARK-31457) spark jdbc read hive created the wrong PreparedStatement

[ https://issues.apache.org/jira/browse/SPARK-31457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] daile updated SPARK-31457: -- Attachment: hivejdbc3.png > spark jdbc read hive created the wrong PreparedStatement > ---

[jira] [Updated] (SPARK-31457) spark jdbc read hive created the wrong PreparedStatement

[ https://issues.apache.org/jira/browse/SPARK-31457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] daile updated SPARK-31457: -- Attachment: hivejdbc2.png > spark jdbc read hive created the wrong PreparedStatement > ---

[jira] [Updated] (SPARK-31457) spark jdbc read hive created the wrong PreparedStatement

[ https://issues.apache.org/jira/browse/SPARK-31457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] daile updated SPARK-31457: -- Attachment: sparkhivejdbc.png > spark jdbc read hive created the wrong PreparedStatement > ---

[jira] [Created] (SPARK-31457) spark jdbc read hive created the wrong PreparedStatement

daile created SPARK-31457: - Summary: spark jdbc read hive created the wrong PreparedStatement Key: SPARK-31457 URL: https://issues.apache.org/jira/browse/SPARK-31457 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31456) If shutdownhook is added with priority Integer.MIN_VALUE, it's supposed to be called the last, but it gets called before other positive priority shutdownhook

Xiaolei Liu created SPARK-31456: --- Summary: If shutdownhook is added with priority Integer.MIN_VALUE, it's supposed to be called the last, but it gets called before other positive priority shutdownhook Key: SPARK-31456

[jira] [Commented] (SPARK-31455) Fix rebasing of not-existed dates/timestamps

[ https://issues.apache.org/jira/browse/SPARK-31455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084501#comment-17084501 ] Wenchen Fan commented on SPARK-31455: - also https://github.com/apache/spark/pull/282

[jira] [Resolved] (SPARK-31455) Fix rebasing of not-existed timestamps

[ https://issues.apache.org/jira/browse/SPARK-31455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31455. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28227 [https://gith

[jira] [Updated] (SPARK-31455) Fix rebasing of not-existed dates/timestamps

[ https://issues.apache.org/jira/browse/SPARK-31455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31455: Summary: Fix rebasing of not-existed dates/timestamps (was: Fix rebasing of not-existed timestamp

[jira] [Created] (SPARK-31455) Fix rebasing of not-existed timestamps

Wenchen Fan created SPARK-31455: --- Summary: Fix rebasing of not-existed timestamps Key: SPARK-31455 URL: https://issues.apache.org/jira/browse/SPARK-31455 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-15616) CatalogRelation should fallback to HDFS size of partitions that are involved in Query if statistics are not available.

[ https://issues.apache.org/jira/browse/SPARK-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-15616: --- Assignee: Hu Fuwang > CatalogRelation should fallback to HDFS size of partitions that are involved

[jira] [Created] (SPARK-31454) An optimized K-Means based on DenseMatrix and GEMM

Xiaochang Wu created SPARK-31454: Summary: An optimized K-Means based on DenseMatrix and GEMM Key: SPARK-31454 URL: https://issues.apache.org/jira/browse/SPARK-31454 Project: Spark Issue Type

[jira] [Resolved] (SPARK-31428) Document Common Table Expression in SQL Reference

[ https://issues.apache.org/jira/browse/SPARK-31428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31428. -- Fix Version/s: 3.0.0 Assignee: Huaxin Gao Resolution: Fixed Resolved b

[jira] [Commented] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084431#comment-17084431 ] Baohe Zhang commented on SPARK-31380: - Did you run your application with Spark 3? I

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Baohe Zhang updated SPARK-31380: Attachment: image-2020-04-15-18-16-18-254.png > Peak Execution Memory Quantile is not displayed in

[jira] [Comment Edited] (SPARK-31236) Spark error while consuming data from Kinesis direct end point

[ https://issues.apache.org/jira/browse/SPARK-31236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084419#comment-17084419 ] Thukarama Prabhu edited comment on SPARK-31236 at 4/15/20, 10:45 PM: -

[jira] [Commented] (SPARK-31236) Spark error while consuming data from Kinesis direct end point

[ https://issues.apache.org/jira/browse/SPARK-31236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084419#comment-17084419 ] Thukarama Prabhu commented on SPARK-31236: -- Looks like this requires major code

[jira] [Commented] (SPARK-31399) Closure cleaner broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084356#comment-17084356 ] Xiao Li commented on SPARK-31399: - [~rednaxelafx] will help this ticket and do more inve

[jira] [Assigned] (SPARK-31399) Closure cleaner broken in Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-31399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-31399: --- Assignee: Kris Mok > Closure cleaner broken in Scala 2.12 > >

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084308#comment-17084308 ] Maxim Gekk commented on SPARK-31423: [~bersprockets] I think we should take the next

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084288#comment-17084288 ] Bruce Robbins commented on SPARK-31423: --- Thanks. It seems we can either - close t

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084234#comment-17084234 ] Wenchen Fan commented on SPARK-31423: - I hope the ORC community can figure this out

[jira] [Commented] (SPARK-31371) FileStreamSource: Decide seen files on the checksum, instead of filename.

[ https://issues.apache.org/jira/browse/SPARK-31371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084223#comment-17084223 ] Gabor Somogyi commented on SPARK-31371: --- There was a similar feature request befor

[jira] [Created] (SPARK-31453) Error while converting JavaRDD to Dataframe

Sachit Sharma created SPARK-31453: - Summary: Error while converting JavaRDD to Dataframe Key: SPARK-31453 URL: https://issues.apache.org/jira/browse/SPARK-31453 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084218#comment-17084218 ] Bruce Robbins commented on SPARK-31423: --- OK, so this is a case of a limitation of

[jira] [Assigned] (SPARK-25440) Dump query execution info to a file

[ https://issues.apache.org/jira/browse/SPARK-25440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25440: --- Assignee: Maxim Gekk > Dump query execution info to a file > --- >

[jira] [Created] (SPARK-31452) Do not create partition spec for 0-size partitions

Wenchen Fan created SPARK-31452: --- Summary: Do not create partition spec for 0-size partitions Key: SPARK-31452 URL: https://issues.apache.org/jira/browse/SPARK-31452 Project: Spark Issue Type:

[jira] [Created] (SPARK-31451) Kafka connector does not retry in case of RetriableException

Chaoran Yu created SPARK-31451: -- Summary: Kafka connector does not retry in case of RetriableException Key: SPARK-31451 URL: https://issues.apache.org/jira/browse/SPARK-31451 Project: Spark Iss

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084086#comment-17084086 ] Thomas Graves commented on SPARK-31437: --- so there are multiple reasons they are ti

[jira] [Commented] (SPARK-30364) The spark-streaming-kafka-0-10_2.11 test cases are failing on ppc64le

[ https://issues.apache.org/jira/browse/SPARK-30364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084033#comment-17084033 ] Nick Hryhoriev commented on SPARK-30364: I have the same issue with the spark ap

[jira] [Resolved] (SPARK-31394) Support for Kubernetes NFS volume mounts

[ https://issues.apache.org/jira/browse/SPARK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31394. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27364 [https://

[jira] [Assigned] (SPARK-31394) Support for Kubernetes NFS volume mounts

[ https://issues.apache.org/jira/browse/SPARK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31394: - Assignee: Seongjin Cho > Support for Kubernetes NFS volume mounts > ---

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083959#comment-17083959 ] Hongze Zhang commented on SPARK-31437: -- Thanks [~tgraves] and I too found some rela

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-31429: - Issue Type: Improvement (was: Bug) > Add additional fields in ExpressionDescription for

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083937#comment-17083937 ] Zhou Jiashuai commented on SPARK-26385: --- [~gsomogyi] There is no unusual log detec

[jira] [Updated] (SPARK-31394) Support for Kubernetes NFS volume mounts

[ https://issues.apache.org/jira/browse/SPARK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31394: -- Affects Version/s: (was: 3) 3.1.0 > Support for Kubernetes NFS volu

[jira] [Updated] (SPARK-31394) Support for Kubernetes NFS volume mounts

[ https://issues.apache.org/jira/browse/SPARK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31394: -- Shepherd: (was: Dongjoon Hyun) > Support for Kubernetes NFS volume mounts >

[jira] [Updated] (SPARK-31394) Support for Kubernetes NFS volume mounts

[ https://issues.apache.org/jira/browse/SPARK-31394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31394: -- Priority: Major (was: Minor) > Support for Kubernetes NFS volume mounts > ---

[jira] [Created] (SPARK-31450) Make ExpressionEncoder thread safe

Herman van Hövell created SPARK-31450: - Summary: Make ExpressionEncoder thread safe Key: SPARK-31450 URL: https://issues.apache.org/jira/browse/SPARK-31450 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-27296) Efficient User Defined Aggregators

[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083901#comment-17083901 ] Wenchen Fan commented on SPARK-27296: - This feature is to speed up UDAF by using Agg

[jira] [Issue Comment Deleted] (SPARK-31226) SizeBasedCoalesce logic error

[ https://issues.apache.org/jira/browse/SPARK-31226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sathyaprakash Govindasamy updated SPARK-31226: -- Comment: was deleted (was: [~angerszhuuu] Could you please share more

[jira] [Commented] (SPARK-31226) SizeBasedCoalesce logic error

[ https://issues.apache.org/jira/browse/SPARK-31226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083890#comment-17083890 ] Sathyaprakash Govindasamy commented on SPARK-31226: --- [~angerszhuuu] Co

[jira] [Updated] (SPARK-31447) DATE_PART functions produces incorrect result

[ https://issues.apache.org/jira/browse/SPARK-31447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sathyaprakash Govindasamy updated SPARK-31447: -- Priority: Minor (was: Major) > DATE_PART functions produces incorrect

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083888#comment-17083888 ] Gabor Somogyi commented on SPARK-26385: --- [~zjiash] Please don't forget to attach f

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083871#comment-17083871 ] Zhou Jiashuai commented on SPARK-26385: --- We use yarn-cluster mode and I will set