[jira] [Created] (SPARK-28893) Support MERGE INTO in DataSource V2

Xianyin Xin created SPARK-28893: --- Summary: Support MERGE INTO in DataSource V2 Key: SPARK-28893 URL: https://issues.apache.org/jira/browse/SPARK-28893 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-28892) Support UPDATE in DataSource V2

[ https://issues.apache.org/jira/browse/SPARK-28892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-28892: Affects Version/s: (was: 2.4.3) 3.0.0 > Support UPDATE in DataSource V2

[jira] [Created] (SPARK-28892) Support UPDATE in DataSource V2

Xianyin Xin created SPARK-28892: --- Summary: Support UPDATE in DataSource V2 Key: SPARK-28892 URL: https://issues.apache.org/jira/browse/SPARK-28892 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-28891) do-release-docker.sh in master does not work for branch-2.3

[ https://issues.apache.org/jira/browse/SPARK-28891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-28891: - Description: According to [~maropu], [do-release-docker.sh|https://github.com/apache/sp

[jira] [Created] (SPARK-28891) do-release-docker.sh in master does not work for branch-2.3

Kazuaki Ishizaki created SPARK-28891: Summary: do-release-docker.sh in master does not work for branch-2.3 Key: SPARK-28891 URL: https://issues.apache.org/jira/browse/SPARK-28891 Project: Spark

[jira] [Assigned] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23519: --- Assignee: hemanth meka > Create View Commands Fails with The view output (col1,col1) conta

[jira] [Resolved] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23519. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25570 [https://gith

[jira] [Created] (SPARK-28890) Upgrade Hive Metastore Client to the latest versions for Hive 3.1

Yuming Wang created SPARK-28890: --- Summary: Upgrade Hive Metastore Client to the latest versions for Hive 3.1 Key: SPARK-28890 URL: https://issues.apache.org/jira/browse/SPARK-28890 Project: Spark

[jira] [Commented] (SPARK-28889) Allow UDTs to define custom casting behavior

[ https://issues.apache.org/jira/browse/SPARK-28889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917341#comment-16917341 ] Zachary S Ennenga commented on SPARK-28889: --- I don't know the intent behind so

[jira] [Updated] (SPARK-28889) Allow UDTs to define custom casting behavior

[ https://issues.apache.org/jira/browse/SPARK-28889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zachary S Ennenga updated SPARK-28889: -- Description: Looking at `org.apache.spark.sql.catalyst.expressions.Cast`, UDTs do not

[jira] [Updated] (SPARK-28889) Allow UDTs to define custom casting behavior

[ https://issues.apache.org/jira/browse/SPARK-28889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zachary S Ennenga updated SPARK-28889: -- Component/s: (was: Spark Core) SQL > Allow UDTs to define custom

[jira] [Commented] (SPARK-28889) Allow UDTs to define custom casting behavior

[ https://issues.apache.org/jira/browse/SPARK-28889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917152#comment-16917152 ] Zachary S Ennenga commented on SPARK-28889: --- I think this change is pretty sim

[jira] [Created] (SPARK-28889) Allow UDTs to define custom casting behavior

Zachary S Ennenga created SPARK-28889: - Summary: Allow UDTs to define custom casting behavior Key: SPARK-28889 URL: https://issues.apache.org/jira/browse/SPARK-28889 Project: Spark Issue

[jira] [Resolved] (SPARK-28888) Implement Dynamic Partition Pruning

[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-2. - Resolution: Duplicate > Implement Dynamic Partition Pruning > --- > >

[jira] [Reopened] (SPARK-11150) Dynamic partition pruning

[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-11150: - > Dynamic partition pruning > - > > Key: SPARK-11150 >

[jira] [Resolved] (SPARK-11150) Dynamic partition pruning

[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-11150. - Resolution: Duplicate > Dynamic partition pruning > - > > Key: S

[jira] [Updated] (SPARK-28888) Implement Dynamic Partition Pruning

[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-2: Target Version/s: 3.0.0 > Implement Dynamic Partition Pruning > --- > >

[jira] [Created] (SPARK-28888) Implement Dynamic Partition Pruning

Maryann Xue created SPARK-2: --- Summary: Implement Dynamic Partition Pruning Key: SPARK-2 URL: https://issues.apache.org/jira/browse/SPARK-2 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-28887) Executor pod status bug with long running sidecar container

jinxing created SPARK-28887: --- Summary: Executor pod status bug with long running sidecar container Key: SPARK-28887 URL: https://issues.apache.org/jira/browse/SPARK-28887 Project: Spark Issue Type

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27992: -- Fix Version/s: 2.4.4 > PySpark socket server should sync with JVM connection thread future > -

[jira] [Updated] (SPARK-28881) toPandas with Arrow should not return a DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28881: -- Fix Version/s: 2.4.4 > toPandas with Arrow should not return a DataFrame when the result size

[jira] [Created] (SPARK-28886) Kubernetes DepsTestsSuite fails on OSX with minikube 1.3.1 due to formatting

holdenk created SPARK-28886: --- Summary: Kubernetes DepsTestsSuite fails on OSX with minikube 1.3.1 due to formatting Key: SPARK-28886 URL: https://issues.apache.org/jira/browse/SPARK-28886 Project: Spark

[jira] [Updated] (SPARK-28871) Some codes in 'Policy for handling multiple watermarks' does not show friendly

[ https://issues.apache.org/jira/browse/SPARK-28871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28871: -- Fix Version/s: 2.4.4 > Some codes in 'Policy for handling multiple watermarks' does not show

[jira] [Updated] (SPARK-28750) Use `--release 8` for javac

[ https://issues.apache.org/jira/browse/SPARK-28750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28750: -- Parent: (was: SPARK-24417) Issue Type: Improvement (was: Sub-task) > Use `--relea

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916878#comment-16916878 ] Dongjoon Hyun commented on SPARK-24417: --- Now, we succeeded to build with JDK8 and

[jira] [Updated] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24663: -- Affects Version/s: 3.0.0 > Flaky test: StreamingContextSuite "stop slow receiver gracefully" >

[jira] [Resolved] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28495. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25581 [https://gith

[jira] [Assigned] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28495: --- Assignee: Gengliang Wang > Introduce ANSI store assignment policy for table insertion > ---

[jira] [Resolved] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27360. --- Fix Version/s: 3.0.0 Resolution: Fixed all subtasks are finished so resolving > Stan

[jira] [Resolved] (SPARK-28414) Standalone worker/master UI updates for Resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-28414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-28414. --- Fix Version/s: 3.0.0 Assignee: wuyi Resolution: Fixed > Standalone worker/ma

[jira] [Resolved] (SPARK-28621) CheckCartesianProducts may throw some error which mismatch generated physical plan

[ https://issues.apache.org/jira/browse/SPARK-28621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28621. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25520 [https://gith

[jira] [Assigned] (SPARK-28621) CheckCartesianProducts may throw some error which mismatch generated physical plan

[ https://issues.apache.org/jira/browse/SPARK-28621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28621: --- Assignee: Weichen Xu > CheckCartesianProducts may throw some error which mismatch generated

[jira] [Resolved] (SPARK-28876) fallBackToHdfs should not support Hive partitioned table

[ https://issues.apache.org/jira/browse/SPARK-28876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28876. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25584 [https://gith

[jira] [Assigned] (SPARK-28876) fallBackToHdfs should not support Hive partitioned table

[ https://issues.apache.org/jira/browse/SPARK-28876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28876: --- Assignee: Yuming Wang > fallBackToHdfs should not support Hive partitioned table >

[jira] [Updated] (SPARK-28885) Follow ANSI store assignment rules in table insertion by default

[ https://issues.apache.org/jira/browse/SPARK-28885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28885: --- Description: When inserting a value into a column with a different data type, Spark will pe

[jira] [Created] (SPARK-28885) Follow ANSI store assignment rules in table insertion by default

Gengliang Wang created SPARK-28885: -- Summary: Follow ANSI store assignment rules in table insertion by default Key: SPARK-28885 URL: https://issues.apache.org/jira/browse/SPARK-28885 Project: Spark

[jira] [Resolved] (SPARK-28871) Some codes in 'Policy for handling multiple watermarks' does not show friendly

[ https://issues.apache.org/jira/browse/SPARK-28871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28871. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25580 [https://github.c

[jira] [Assigned] (SPARK-28871) Some codes in 'Policy for handling multiple watermarks' does not show friendly

[ https://issues.apache.org/jira/browse/SPARK-28871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28871: - Assignee: chaiyongqiang > Some codes in 'Policy for handling multiple watermarks' does not show

[jira] [Resolved] (SPARK-28747) merge the two data source v2 fallback configs

[ https://issues.apache.org/jira/browse/SPARK-28747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28747. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25465 [https://gith

[jira] [Resolved] (SPARK-28691) Add Java/Scala DirectKerberizedKafkaWordCount examples

[ https://issues.apache.org/jira/browse/SPARK-28691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28691. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25412 [https://gi

[jira] [Assigned] (SPARK-28691) Add Java/Scala DirectKerberizedKafkaWordCount examples

[ https://issues.apache.org/jira/browse/SPARK-28691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28691: Assignee: hong dongdong > Add Java/Scala DirectKerberizedKafkaWordCount examples > --

[jira] [Commented] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916579#comment-16916579 ] Jungtaek Lim commented on SPARK-24663: -- Hit this again. [https://amplab.cs.berkele

[jira] [Commented] (SPARK-26046) Add a way for StreamingQueryManager to remove all listeners

[ https://issues.apache.org/jira/browse/SPARK-26046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916570#comment-16916570 ] Gabor Somogyi commented on SPARK-26046: --- >From the author: {quote}I think I'm tryi

[jira] [Resolved] (SPARK-26046) Add a way for StreamingQueryManager to remove all listeners

[ https://issues.apache.org/jira/browse/SPARK-26046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-26046. --- Resolution: Not A Problem > Add a way for StreamingQueryManager to remove all listeners > --

[jira] [Commented] (SPARK-28884) [Spark]Spark driver cores is showing 0 instead of 1 in UI for cluster mode deployment

[ https://issues.apache.org/jira/browse/SPARK-28884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916530#comment-16916530 ] Sandeep Katta commented on SPARK-28884: --- [~abhishek.akg] thanks for raising this i

[jira] [Resolved] (SPARK-28881) toPandas with Arrow should not return a DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28881. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25594 [https://gi

[jira] [Assigned] (SPARK-28881) toPandas with Arrow should not return a DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28881: Assignee: Hyukjin Kwon > toPandas with Arrow should not return a DataFrame when the resul

[jira] [Commented] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916478#comment-16916478 ] Dongjoon Hyun commented on SPARK-27992: --- I updated the issue type from `Improvemen

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27992: -- Issue Type: Bug (was: Improvement) > PySpark socket server should sync with JVM connection th

[jira] [Updated] (SPARK-28759) Upgrade scala-maven-plugin to 4.1.1

[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28759: -- Parent: (was: SPARK-24417) Issue Type: Improvement (was: Sub-task) > Upgrade scal

[jira] [Commented] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916468#comment-16916468 ] Dongjoon Hyun commented on SPARK-27992: --- Thanks! > PySpark socket server should s

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27992: - Labels: correctness (was: ) > PySpark socket server should sync with JVM connection thread futu

[jira] [Commented] (SPARK-28881) toPandas with Arrow should not return a DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916464#comment-16916464 ] Hyukjin Kwon commented on SPARK-28881: -- I did that in SPARK-27992. Technically it s

[jira] [Updated] (SPARK-28881) toPandas with Arrow should not return a DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28881: -- Summary: toPandas with Arrow should not return a DataFrame when the result size exceeds `spark

[jira] [Updated] (SPARK-28884) [Spark]Spark driver cores is showing 0 instead of 1 in UI for cluster mode deployment

[ https://issues.apache.org/jira/browse/SPARK-28884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-28884: - Attachment: Core1.png Core0.png > [Spark]Spark driver cores is s

[jira] [Updated] (SPARK-28881) toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28881: -- Affects Version/s: (was: 2.3.3) > toPandas with Arrow returns an empty DataFrame when the

[jira] [Comment Edited] (SPARK-28881) toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916454#comment-16916454 ] Dongjoon Hyun edited comment on SPARK-28881 at 8/27/19 7:03 AM: --

[jira] [Commented] (SPARK-28881) toPandas with Arrow returns an empty DataFrame when the result size exceeds `spark.driver.maxResultSize`

[ https://issues.apache.org/jira/browse/SPARK-28881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916454#comment-16916454 ] Dongjoon Hyun commented on SPARK-28881: --- Could you update the bug type? `Tests`. A