[jira] [Issue Comment Deleted] (SPARK-31220) repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.sql.adaptive.enabled

2020-03-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31220: Comment: was deleted (was: I'm working on.) > repartition obeys spark.sql.adaptive.coalescePartit

[jira] [Resolved] (SPARK-31229) Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast about null <> complex types

2020-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31229. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27990 [https://gi

[jira] [Assigned] (SPARK-31229) Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast about null <> complex types

2020-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31229: Assignee: Hyukjin Kwon > Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast

[jira] [Resolved] (SPARK-31205) support string literal as the second argument of date_add/date_sub functions

2020-03-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31205. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27965 [https://gith

[jira] [Commented] (SPARK-31209) Not compatible with new version of scalatest (3.1.0 and above)

2020-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065283#comment-17065283 ] Hyukjin Kwon commented on SPARK-31209: -- Please go ahead and upgrade if you're inter

[jira] [Commented] (SPARK-31210) An issue for Spark SQL LIKE-with-ESCAPE clause

2020-03-23 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065239#comment-17065239 ] jiaan.geng commented on SPARK-31210: Yes, spark 3.0.0-preview2 not contains this fix

[jira] [Updated] (SPARK-31223) Update py code to generate data in testsuites

2020-03-23 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-31223: --- Summary: Update py code to generate data in testsuites (was: Update py code to generate dates in te

[jira] [Updated] (SPARK-27963) Allow dynamic allocation without an external shuffle service

2020-03-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27963: -- Labels: release-notes (was: ) > Allow dynamic allocation without an external shuffle service

[jira] [Resolved] (SPARK-31203) Upgrade derby to 10.14.2.0 from 10.12.1.1

2020-03-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31203. --- Resolution: Later > Upgrade derby to 10.14.2.0 from 10.12.1.1 >

[jira] [Comment Edited] (SPARK-31210) An issue for Spark SQL LIKE-with-ESCAPE clause

2020-03-23 Thread Mingli Rui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064942#comment-17064942 ] Mingli Rui edited comment on SPARK-31210 at 3/23/20, 4:53 PM:

[jira] [Commented] (SPARK-31210) An issue for Spark SQL LIKE-with-ESCAPE clause

2020-03-23 Thread Mingli Rui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064942#comment-17064942 ] Mingli Rui commented on SPARK-31210: Thanks for you investigation! I agree this issu

[jira] [Comment Edited] (SPARK-31191) Spark SQL and hive metastore are incompatible

2020-03-23 Thread leishuiyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064808#comment-17064808 ] leishuiyu edited comment on SPARK-31191 at 3/23/20, 1:41 PM: -

[jira] [Comment Edited] (SPARK-31191) Spark SQL and hive metastore are incompatible

2020-03-23 Thread leishuiyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064808#comment-17064808 ] leishuiyu edited comment on SPARK-31191 at 3/23/20, 1:40 PM: -

[jira] [Commented] (SPARK-31191) Spark SQL and hive metastore are incompatible

2020-03-23 Thread leishuiyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064808#comment-17064808 ] leishuiyu commented on SPARK-31191: --- when edit conf/spark-defaults.conf  ,this error s

[jira] [Updated] (SPARK-31191) Spark SQL and hive metastore are incompatible

2020-03-23 Thread leishuiyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leishuiyu updated SPARK-31191: -- Attachment: image-2020-03-23-21-37-17-663.png > Spark SQL and hive metastore are incompatible > --

[jira] [Created] (SPARK-31230) use statement plans in DataFrameWriter(V2)

2020-03-23 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-31230: --- Summary: use statement plans in DataFrameWriter(V2) Key: SPARK-31230 URL: https://issues.apache.org/jira/browse/SPARK-31230 Project: Spark Issue Type: Improvem

[jira] [Created] (SPARK-31229) Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast about null <> complex types

2020-03-23 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-31229: Summary: Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast about null <> complex types Key: SPARK-31229 URL: https://issues.apache.org/jira/browse/SPARK-31229

[jira] [Updated] (SPARK-31229) Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast about null <> complex types

2020-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31229: - Priority: Minor (was: Major) > Add unit tests TypeCoercion.findTypeForComplex and Cast.canCast

[jira] [Created] (SPARK-31228) Add version information to the configuration of Kafka

2020-03-23 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-31228: -- Summary: Add version information to the configuration of Kafka Key: SPARK-31228 URL: https://issues.apache.org/jira/browse/SPARK-31228 Project: Spark Issue Type:

[jira] [Created] (SPARK-31227) Non-nullable null type should not coerce to nullable type

2020-03-23 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-31227: Summary: Non-nullable null type should not coerce to nullable type Key: SPARK-31227 URL: https://issues.apache.org/jira/browse/SPARK-31227 Project: Spark Iss

[jira] [Updated] (SPARK-31227) Non-nullable null type should not coerce to nullable type

2020-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31227: - Affects Version/s: (was: 3.0.0) 3.1.0 > Non-nullable null type should

[jira] [Closed] (SPARK-31194) spark sql runs successfully with query not specifying condition next to where

2020-03-23 Thread Ayoub Omari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayoub Omari closed SPARK-31194. --- Where works as an alias to tableName in this case  > spark sql runs successfully with query not specify

[jira] [Comment Edited] (SPARK-31194) spark sql runs successfully with query not specifying condition next to where

2020-03-23 Thread Ayoub Omari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064692#comment-17064692 ] Ayoub Omari edited comment on SPARK-31194 at 3/23/20, 10:28 AM: --

[jira] [Updated] (SPARK-31219) YarnShuffleService doesn't close idle netty channel

2020-03-23 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated SPARK-31219: --- Description: Recently, we find our YarnShuffleService has a lot of [half-open connections|https://b

[jira] [Created] (SPARK-31226) SizeBasedCoalesce logic error

2020-03-23 Thread angerszhu (Jira)
angerszhu created SPARK-31226: - Summary: SizeBasedCoalesce logic error Key: SPARK-31226 URL: https://issues.apache.org/jira/browse/SPARK-31226 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-31226) SizeBasedCoalesce logic error

2020-03-23 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-31226: -- Description: In spark UT,  SizeBasedCoalecse's logic is wrong > SizeBasedCoalesce logic error >

[jira] [Updated] (SPARK-31190) ScalaReflection should not erasure user defined AnyVal type

2020-03-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31190: Summary: ScalaReflection should not erasure user defined AnyVal type (was: ScalaReflection should

[jira] [Resolved] (SPARK-31190) ScalaReflection should erasure non user defined AnyVal type

2020-03-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31190. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27959 [https://gith

[jira] [Assigned] (SPARK-31190) ScalaReflection should erasure non user defined AnyVal type

2020-03-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31190: --- Assignee: wuyi > ScalaReflection should erasure non user defined AnyVal type >

[jira] [Updated] (SPARK-31220) repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.sql.adaptive.enabled

2020-03-23 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31220: Summary: repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.s

[jira] [Created] (SPARK-31225) Override `sql` method for OuterReference

2020-03-23 Thread Kent Yao (Jira)
Kent Yao created SPARK-31225: Summary: Override `sql` method for OuterReference Key: SPARK-31225 URL: https://issues.apache.org/jira/browse/SPARK-31225 Project: Spark Issue Type: Improvement