[jira] [Commented] (SPARK-28482) Data incomplete when using pandas udf in pyspark

[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891638#comment-16891638 ] jiangyu commented on SPARK-28482: - hi  [~dongjoon] , I have tested this in spark 2.3.3 ,

[jira] [Commented] (SPARK-28494) Expose CalendarIntervalType and CalendarInterval in Spark

[ https://issues.apache.org/jira/browse/SPARK-28494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891635#comment-16891635 ] Yuming Wang commented on SPARK-28494: - Do we need to support save interval type to t

[jira] [Updated] (SPARK-13677) Support Tree-Based Feature Transformation for ML

[ https://issues.apache.org/jira/browse/SPARK-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-13677: - Description: It would be nice to be able to use RF and GBT for feature transformation: First fi

[jira] [Updated] (SPARK-13677) Support Tree-Based Feature Transformation for ML

[ https://issues.apache.org/jira/browse/SPARK-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-13677: - Description: It would be nice to be able to use RF and GBT for feature transformation: First fi

[jira] [Updated] (SPARK-26074) AsyncEventQueue.stop hangs when eventQueue is full

[ https://issues.apache.org/jira/browse/SPARK-26074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26074: -- Fix Version/s: (was: 2.3.1) (was: 2.4.0) > AsyncEventQueue.stop han

[jira] [Closed] (SPARK-26074) AsyncEventQueue.stop hangs when eventQueue is full

[ https://issues.apache.org/jira/browse/SPARK-26074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26074. - > AsyncEventQueue.stop hangs when eventQueue is full > -

[jira] [Updated] (SPARK-27727) Asynchronous ElementStore cleanup should have only one pending cleanup per class

[ https://issues.apache.org/jira/browse/SPARK-27727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27727: -- Fix Version/s: (was: 3.0.0) > Asynchronous ElementStore cleanup should have only one pendi

[jira] [Updated] (SPARK-27728) Address thread-safety of InMemoryStore and ElementTrackingStores.

[ https://issues.apache.org/jira/browse/SPARK-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27728: -- Fix Version/s: (was: 3.0.0) > Address thread-safety of InMemoryStore and ElementTrackingSt

[jira] [Closed] (SPARK-21157) Report Total Memory Used by Spark Executors

[ https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21157. - > Report Total Memory Used by Spark Executors > --- > >

[jira] [Updated] (SPARK-21157) Report Total Memory Used by Spark Executors

[ https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21157: -- Fix Version/s: (was: 3.0.0) > Report Total Memory Used by Spark Executors > --

[jira] [Updated] (SPARK-27729) Extract deletion of the summaries from the stage deletion loop

[ https://issues.apache.org/jira/browse/SPARK-27729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27729: -- Fix Version/s: (was: 3.0.0) > Extract deletion of the summaries from the stage deletion lo

[jira] [Closed] (SPARK-27728) Address thread-safety of InMemoryStore and ElementTrackingStores.

[ https://issues.apache.org/jira/browse/SPARK-27728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27728. - > Address thread-safety of InMemoryStore and ElementTrackingStores. > --

[jira] [Closed] (SPARK-27727) Asynchronous ElementStore cleanup should have only one pending cleanup per class

[ https://issues.apache.org/jira/browse/SPARK-27727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27727. - > Asynchronous ElementStore cleanup should have only one pending cleanup per > class >

[jira] [Closed] (SPARK-27729) Extract deletion of the summaries from the stage deletion loop

[ https://issues.apache.org/jira/browse/SPARK-27729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27729. - > Extract deletion of the summaries from the stage deletion loop > -

[jira] [Closed] (SPARK-27731) Cleanup some non-compile time type checking and exception handling

[ https://issues.apache.org/jira/browse/SPARK-27731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27731. - > Cleanup some non-compile time type checking and exception handling > -

[jira] [Closed] (SPARK-27848) AppVeyor change to latest R version (3.6.0)

[ https://issues.apache.org/jira/browse/SPARK-27848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27848. - > AppVeyor change to latest R version (3.6.0) > --- > >

[jira] [Closed] (SPARK-27730) Add support for removeAllKeys

[ https://issues.apache.org/jira/browse/SPARK-27730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27730. - > Add support for removeAllKeys > - > > Key: SPARK-27730

[jira] [Closed] (SPARK-27685) `union` doesn't promote non-nullable columns of struct to nullable

[ https://issues.apache.org/jira/browse/SPARK-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27685. - > `union` doesn't promote non-nullable columns of struct to nullable > -

[jira] [Updated] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26845: -- Labels: (was: correctness) > Avro to_avro from_avro roundtrip fails if data type is string >

[jira] [Updated] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26845: -- Priority: Major (was: Critical) > Avro to_avro from_avro roundtrip fails if data type is stri

[jira] [Resolved] (SPARK-28390) Convert and port 'pgSQL/select_having.sql' into UDF test base

[ https://issues.apache.org/jira/browse/SPARK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28390. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25161 [https://gi

[jira] [Assigned] (SPARK-28390) Convert and port 'pgSQL/select_having.sql' into UDF test base

[ https://issues.apache.org/jira/browse/SPARK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28390: Assignee: Shivu Sondur > Convert and port 'pgSQL/select_having.sql' into UDF test base >

[jira] [Closed] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26845. - > Avro to_avro from_avro roundtrip fails if data type is string > --

[jira] [Commented] (SPARK-28482) Data incomplete when using pandas udf in pyspark

[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891618#comment-16891618 ] Dongjoon Hyun commented on SPARK-28482: --- Thank you for reporting, [~jiangyu1211].

[jira] [Updated] (SPARK-28493) Expose and support calendar interval type in SparkR

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Summary: Expose and support calendar interval type in SparkR (was: To expose Calendar Interval

[jira] [Updated] (SPARK-28493) Expose and support calendar interval type in SparkR

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Component/s: SparkR > Expose and support calendar interval type in SparkR >

[jira] [Updated] (SPARK-28493) Expose and support calendar interval type in SparkR

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Component/s: (was: SparkR) > Expose and support calendar interval type in SparkR > -

[jira] [Updated] (SPARK-28492) Expose and support calendar interval type in PySpark

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28492: - Component/s: (was: SQL) PySpark > Expose and support calendar interval type

[jira] [Updated] (SPARK-28491) Expose and support calendar interval type in SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28491: - Summary: Expose and support calendar interval type in SparkSQL (was: To do Sql parser side code

[jira] [Updated] (SPARK-28492) Expose and support calendar interval type in PySpark

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28492: - Summary: Expose and support calendar interval type in PySpark (was: To expose calendar interval

[jira] [Updated] (SPARK-28493) To expose Calendar Interval type in R

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Description: Reference : [https://github.com/apache/spark/pull/25022] See https://github.com/a

[jira] [Updated] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25590: -- Fix Version/s: (was: 3.0.0) > kubernetes-model-2.0.0.jar masks default Spark logging confi

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891614#comment-16891614 ] Dongjoon Hyun commented on SPARK-25590: --- I removed the `Fixed Version: 3.0.0` acco

[jira] [Updated] (SPARK-28492) To expose calendar interval type in python.

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28492: - Description: Reference : [https://github.com/apache/spark/pull/25022 ] See https://github.com/a

[jira] [Updated] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval type.

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28491: - Description: Reference : [https://github.com/apache/spark/pull/25022] See [https://github.com/ap

[jira] [Updated] (SPARK-28492) To expose calendar interval type in python.

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28492: - Description: Reference : [https://github.com/apache/spark/pull/25022] See https://github.com/apa

[jira] [Updated] (SPARK-28493) To expose Calendar Interval type in R

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-28494 > To expose Calendar Interv

[jira] [Updated] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval type.

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28491: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-28494 > To do Sql parser side cod

[jira] [Updated] (SPARK-26683) Incorrect value of "internal.metrics.input.recordsRead" when reading from temp hive table backed by HDFS file

[ https://issues.apache.org/jira/browse/SPARK-26683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26683: -- Fix Version/s: (was: 3.0.0) > Incorrect value of "internal.metrics.input.recordsRead" when

[jira] [Updated] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval type.

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28491: - Description: Reference : [https://github.com/apache/spark/pull/25022 ]See [https://github.com/a

[jira] [Updated] (SPARK-28492) To expose calendar interval type in python.

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28492: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-28494 > To expose calendar interv

[jira] [Updated] (SPARK-28494) Expose CalendarIntervalType and CalendarInterval in Spark

[ https://issues.apache.org/jira/browse/SPARK-28494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28494: - Issue Type: Umbrella (was: Bug) > Expose CalendarIntervalType and CalendarInterval in Spark > -

[jira] [Updated] (SPARK-27513) Spark tarball with binaries should have files owned by uid 0

[ https://issues.apache.org/jira/browse/SPARK-27513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27513: -- Fix Version/s: (was: 3.0.0) > Spark tarball with binaries should have files owned by uid 0

[jira] [Updated] (SPARK-24695) Move `CalendarInterval` to org.apache.spark.sql.types package

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24695: - Issue Type: Sub-task (was: Bug) Parent: SPARK-28494 > Move `CalendarInterval` to org.ap

[jira] [Resolved] (SPARK-27513) Spark tarball with binaries should have files owned by uid 0

[ https://issues.apache.org/jira/browse/SPARK-27513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27513. --- Resolution: Won't Do > Spark tarball with binaries should have files owned by uid 0 > --

[jira] [Created] (SPARK-28494) Expose CalendarIntervalType and CalendarInterval in Spark

Hyukjin Kwon created SPARK-28494: Summary: Expose CalendarIntervalType and CalendarInterval in Spark Key: SPARK-28494 URL: https://issues.apache.org/jira/browse/SPARK-28494 Project: Spark Iss

[jira] [Updated] (SPARK-24695) Move `CalendarInterval` to org.apache.spark.sql.types package

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24695: - Summary: Move `CalendarInterval` to org.apache.spark.sql.types package (was: Unable to return c

[jira] [Updated] (SPARK-24695) Unable to return calendar interval from udf

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24695: - Description: When i am trying to write an udf which returns calendar interval type, i am gettin

[jira] [Updated] (SPARK-26397) Driver-side only metrics support

[ https://issues.apache.org/jira/browse/SPARK-26397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26397: -- Fix Version/s: (was: 3.0.0) > Driver-side only metrics support > -

[jira] [Updated] (SPARK-28493) To expose Calendar Interval type in R

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28493: - Summary: To expose Calendar Interval type in R (was: To expose expose Calendar Interval type in

[jira] [Commented] (SPARK-27100) Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891610#comment-16891610 ] Dongjoon Hyun commented on SPARK-27100: --- Hi, [~parthc]. I added you to the Apache

[jira] [Updated] (SPARK-27100) Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27100: -- Priority: Critical (was: Major) > Use `Array` instead of `Seq` in `FilePartition` to prevent

[jira] [Updated] (SPARK-27100) Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27100: -- Summary: Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError (was:

[jira] [Assigned] (SPARK-27100) Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27100: - Assignee: Parth Chandra > Use `Array` instead of `Seq` in `FilePartition` to prevent St

[jira] [Commented] (SPARK-28481) More expressions should extend NullIntolerant

[ https://issues.apache.org/jira/browse/SPARK-28481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891609#comment-16891609 ] Takeshi Yamamuro commented on SPARK-28481: -- > I'd propose to add "is this null-

[jira] [Updated] (SPARK-28493) To expose expose Calendar Interval type in R

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Garg updated SPARK-28493: -- Summary: To expose expose Calendar Interval type in R (was: To expose exposer Calendar Interv

[jira] [Created] (SPARK-28493) To expose exposer Calendar Interval type in R

Priyanka Garg created SPARK-28493: - Summary: To expose exposer Calendar Interval type in R Key: SPARK-28493 URL: https://issues.apache.org/jira/browse/SPARK-28493 Project: Spark Issue Type: I

[jira] [Commented] (SPARK-28493) To expose exposer Calendar Interval type in R

[ https://issues.apache.org/jira/browse/SPARK-28493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891608#comment-16891608 ] Priyanka Garg commented on SPARK-28493: --- I'll start working on this once [https://

[jira] [Commented] (SPARK-28492) To expose calendar interval type in python.

[ https://issues.apache.org/jira/browse/SPARK-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891607#comment-16891607 ] Priyanka Garg commented on SPARK-28492: --- I'll start working on this once [https://

[jira] [Created] (SPARK-28492) To expose calendar interval type in python.

Priyanka Garg created SPARK-28492: - Summary: To expose calendar interval type in python. Key: SPARK-28492 URL: https://issues.apache.org/jira/browse/SPARK-28492 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval type.

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyanka Garg updated SPARK-28491: -- Summary: To do Sql parser side code changes for exposing Calendar Interval type. (was: To do

[jira] [Commented] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval.

[ https://issues.apache.org/jira/browse/SPARK-28491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891606#comment-16891606 ] Priyanka Garg commented on SPARK-28491: --- I'll start working on this once [https://

[jira] [Created] (SPARK-28491) To do Sql parser side code changes for exposing Calendar Interval.

Priyanka Garg created SPARK-28491: - Summary: To do Sql parser side code changes for exposing Calendar Interval. Key: SPARK-28491 URL: https://issues.apache.org/jira/browse/SPARK-28491 Project: Spark

[jira] [Updated] (SPARK-28490) Support `TIME` type in Spark

[ https://issues.apache.org/jira/browse/SPARK-28490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu, Lipeng updated SPARK-28490: Issue Type: Sub-task (was: Bug) Parent: SPARK-27764 > Support `TIME` type in Spark >

[jira] [Created] (SPARK-28490) Support `TIME` type in Spark

Zhu, Lipeng created SPARK-28490: --- Summary: Support `TIME` type in Spark Key: SPARK-28490 URL: https://issues.apache.org/jira/browse/SPARK-28490 Project: Spark Issue Type: Bug Componen

[jira] [Resolved] (SPARK-24080) Update the nullability of Filter output based on inferred predicates

[ https://issues.apache.org/jira/browse/SPARK-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24080. -- Resolution: Duplicate > Update the nullability of Filter output based on inferred pred

[jira] [Commented] (SPARK-24079) Update the nullability of Join output based on inferred predicates

[ https://issues.apache.org/jira/browse/SPARK-24079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891584#comment-16891584 ] Takeshi Yamamuro commented on SPARK-24079: -- Yea, I think SPARK-24080 is the sam

[jira] [Commented] (SPARK-27931) Accept 'on' and 'off' as input for boolean data type

[ https://issues.apache.org/jira/browse/SPARK-27931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891581#comment-16891581 ] Takeshi Yamamuro commented on SPARK-27931: -- I feel this has a lower priority...

[jira] [Updated] (SPARK-28471) Formatting dates with negative years

[ https://issues.apache.org/jira/browse/SPARK-28471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28471: Affects Version/s: (was: 2.4.3) 3.0.0 > Formatting dates with negative

[jira] [Resolved] (SPARK-28467) Tests failed if there are not enough executors up before running

[ https://issues.apache.org/jira/browse/SPARK-28467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28467. --- Resolution: Invalid Please see the discussion on the PR. > Tests failed if there are not en

[jira] [Assigned] (SPARK-28435) Support cast StringType to IntervalType for SQL interface

[ https://issues.apache.org/jira/browse/SPARK-28435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28435: - Assignee: Yuming Wang > Support cast StringType to IntervalType for SQL interface > ---

[jira] [Updated] (SPARK-28435) Support accepting the interval keyword in the schema string

[ https://issues.apache.org/jira/browse/SPARK-28435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28435: -- Summary: Support accepting the interval keyword in the schema string (was: Support cast Strin

[jira] [Resolved] (SPARK-28435) Support cast StringType to IntervalType for SQL interface

[ https://issues.apache.org/jira/browse/SPARK-28435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28435. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25189 [https://

[jira] [Updated] (SPARK-28482) Data incomplete when using pandas udf in pyspark

[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiangyu updated SPARK-28482: Description: Hi,   Since Spark 2.3.x, pandas udf has been introduced as default ser/des method when usi

[jira] [Resolved] (SPARK-27168) Add docker integration test for MsSql Server

[ https://issues.apache.org/jira/browse/SPARK-27168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27168. --- Resolution: Fixed Assignee: Zhu, Lipeng Fix Version/s: 3.0.0

[jira] [Updated] (SPARK-27159) Update MsSqlServer dialect handling of BLOB type

[ https://issues.apache.org/jira/browse/SPARK-27159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27159: -- Affects Version/s: 2.4.3 > Update MsSqlServer dialect handling of BLOB type >

[jira] [Updated] (SPARK-27159) Update MsSqlServer dialect handling of BLOB type

[ https://issues.apache.org/jira/browse/SPARK-27159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27159: -- Fix Version/s: 2.4.4 > Update MsSqlServer dialect handling of BLOB type >

[jira] [Commented] (SPARK-27931) Accept 'on' and 'off' as input for boolean data type

[ https://issues.apache.org/jira/browse/SPARK-27931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891530#comment-16891530 ] Yuming Wang commented on SPARK-27931: - [~younggyuchun] Please go ahead. But I'm not

[jira] [Updated] (SPARK-27931) Accept 'on' and 'off' as input for boolean data type

[ https://issues.apache.org/jira/browse/SPARK-27931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27931: Description: This ticket contains three things: 1. Accept 'on' and 'off' as input for boolean dat

[jira] [Resolved] (SPARK-28421) SparseVector.apply performance optimization

[ https://issues.apache.org/jira/browse/SPARK-28421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28421. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25178 [https://github.c

[jira] [Assigned] (SPARK-28421) SparseVector.apply performance optimization

[ https://issues.apache.org/jira/browse/SPARK-28421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28421: - Assignee: zhengruifeng > SparseVector.apply performance optimization >

[jira] [Assigned] (SPARK-27234) Continuous Streaming does not support python UDFs

[ https://issues.apache.org/jira/browse/SPARK-27234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27234: Assignee: Hyukjin Kwon > Continuous Streaming does not support python UDFs >

[jira] [Resolved] (SPARK-27234) Continuous Streaming does not support python UDFs

[ https://issues.apache.org/jira/browse/SPARK-27234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27234. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24946 [https://gi

[jira] [Resolved] (SPARK-28391) Convert and port 'pgSQL/select_implicit.sql' into UDF test base

[ https://issues.apache.org/jira/browse/SPARK-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28391. -- Resolution: Fixed Fix Version/s: 3.0.0 Fixed at [https://github.com/apache/spark/pull/2

[jira] [Assigned] (SPARK-28391) Convert and port 'pgSQL/select_implicit.sql' into UDF test base

[ https://issues.apache.org/jira/browse/SPARK-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28391: Assignee: Udbhav Agrawal > Convert and port 'pgSQL/select_implicit.sql' into UDF test bas

[jira] [Updated] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

[ https://issues.apache.org/jira/browse/SPARK-28489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28489: -- Labels: correctness dataloss (was: ) > KafkaOffsetRangeCalculator.getRanges may drop offsets

[jira] [Created] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

Shixiong Zhu created SPARK-28489: Summary: KafkaOffsetRangeCalculator.getRanges may drop offsets Key: SPARK-28489 URL: https://issues.apache.org/jira/browse/SPARK-28489 Project: Spark Issue T

[jira] [Commented] (SPARK-28484) spark-submit uses wrong SPARK_HOME with deploy-mode "cluster"

[ https://issues.apache.org/jira/browse/SPARK-28484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891482#comment-16891482 ] Shivu Sondur commented on SPARK-28484: -- i will check this issue > spark-submit use

[jira] [Assigned] (SPARK-28473) Build command in README should start with ./

[ https://issues.apache.org/jira/browse/SPARK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28473: - Assignee: Douglas Colkitt > Build command in README should start with ./ >

[jira] [Resolved] (SPARK-28473) Build command in README should start with ./

[ https://issues.apache.org/jira/browse/SPARK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28473. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25231 [https://

[jira] [Created] (SPARK-28488) Race in k8s scheduler shutdown can lead to misleading exceptions.

Marcelo Vanzin created SPARK-28488: -- Summary: Race in k8s scheduler shutdown can lead to misleading exceptions. Key: SPARK-28488 URL: https://issues.apache.org/jira/browse/SPARK-28488 Project: Spark

[jira] [Created] (SPARK-28487) K8S pod allocator behaves poorly with dynamic allocation

Marcelo Vanzin created SPARK-28487: -- Summary: K8S pod allocator behaves poorly with dynamic allocation Key: SPARK-28487 URL: https://issues.apache.org/jira/browse/SPARK-28487 Project: Spark

[jira] [Commented] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891377#comment-16891377 ] Xiao Li commented on SPARK-28486: - [~Ngone51] Please submit a PR to address this issue.

[jira] [Deleted] (SPARK-28485) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li deleted SPARK-28485: > PythonBroadcast may delete the broadcast file while a Python worker still > needs it > ---

[jira] [Assigned] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-28486: --- Assignee: wuyi > PythonBroadcast may delete the broadcast file while a Python worker still > needs

[jira] [Updated] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28486: - Issue Type: Bug (was: New Feature) > PythonBroadcast may delete the broadcast file while a Pyth

[jira] [Created] (SPARK-28486) PythonBroadcast may delete the broadcast file while a Python worker still needs it

Shixiong Zhu created SPARK-28486: Summary: PythonBroadcast may delete the broadcast file while a Python worker still needs it Key: SPARK-28486 URL: https://issues.apache.org/jira/browse/SPARK-28486 Pr

[jira] [Commented] (SPARK-27931) Accept 'on' and 'off' as input for boolean data type

[ https://issues.apache.org/jira/browse/SPARK-27931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891366#comment-16891366 ] YoungGyu Chun commented on SPARK-27931: --- hi [~yumwang], are you working on this?

[jira] [Updated] (SPARK-28485) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-28485: Issue Type: Bug (was: Improvement) > PythonBroadcast may delete the broadcast file while a Python worker

[jira] [Updated] (SPARK-28485) PythonBroadcast may delete the broadcast file while a Python worker still needs it

[ https://issues.apache.org/jira/browse/SPARK-28485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-28485: Description: How to reproduce: Run "bin/pyspark --master local[1,1] --conf spark.memory.fraction=0.0001"

  1   2   >