[jira] [Created] (SPARK-35820) Support cast between different DayTimeIntervalType

angerszhu created SPARK-35820: - Summary: Support cast between different DayTimeIntervalType Key: SPARK-35820 URL: https://issues.apache.org/jira/browse/SPARK-35820 Project: Spark Issue Type: Sub-

[jira] [Commented] (SPARK-35820) Support cast between different DayTimeIntervalType

[ https://issues.apache.org/jira/browse/SPARK-35820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365843#comment-17365843 ] angerszhu commented on SPARK-35820: --- Working on this > Support cast between different

[jira] [Updated] (SPARK-35819) Support cast between different field YearMonthIntervalType

[ https://issues.apache.org/jira/browse/SPARK-35819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-35819: -- Description: Support cast between YearMonthIntervalType  year/tear  year/month month/month > Support

[jira] [Created] (SPARK-35819) Support cast between different field YearMonthIntervalType

angerszhu created SPARK-35819: - Summary: Support cast between different field YearMonthIntervalType Key: SPARK-35819 URL: https://issues.apache.org/jira/browse/SPARK-35819 Project: Spark Issue Ty

[jira] [Commented] (SPARK-35772) Check all year-month interval types in HiveInspectors tests

[ https://issues.apache.org/jira/browse/SPARK-35772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365840#comment-17365840 ] Apache Spark commented on SPARK-35772: -- User 'AngersZh' has created a pull requ

[jira] [Assigned] (SPARK-35772) Check all year-month interval types in HiveInspectors tests

[ https://issues.apache.org/jira/browse/SPARK-35772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35772: Assignee: Apache Spark > Check all year-month interval types in HiveInspectors tests > --

[jira] [Assigned] (SPARK-35772) Check all year-month interval types in HiveInspectors tests

[ https://issues.apache.org/jira/browse/SPARK-35772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35772: Assignee: (was: Apache Spark) > Check all year-month interval types in HiveInspectors

[jira] [Assigned] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35817: Assignee: Apache Spark > Queries against wide Avro tables can be slow > -

[jira] [Assigned] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35817: Assignee: (was: Apache Spark) > Queries against wide Avro tables can be slow > --

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365835#comment-17365835 ] Apache Spark commented on SPARK-35817: -- User 'bersprockets' has created a pull requ

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365834#comment-17365834 ] Apache Spark commented on SPARK-35817: -- User 'bersprockets' has created a pull requ

[jira] [Resolved] (SPARK-35708) Add BaseTest for DataTypeOps

[ https://issues.apache.org/jira/browse/SPARK-35708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-35708. --- Fix Version/s: 3.2.0 Assignee: Yikun Jiang Resolution: Fixed Issue resolved

[jira] [Commented] (SPARK-35470) Enable disallow_untyped_defs mypy check for pyspark.pandas.base.

[ https://issues.apache.org/jira/browse/SPARK-35470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365830#comment-17365830 ] Apache Spark commented on SPARK-35470: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-35470) Enable disallow_untyped_defs mypy check for pyspark.pandas.base.

[ https://issues.apache.org/jira/browse/SPARK-35470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365829#comment-17365829 ] Apache Spark commented on SPARK-35470: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-35470) Enable disallow_untyped_defs mypy check for pyspark.pandas.base.

[ https://issues.apache.org/jira/browse/SPARK-35470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35470: Assignee: (was: Apache Spark) > Enable disallow_untyped_defs mypy check for pyspark.p

[jira] [Assigned] (SPARK-35470) Enable disallow_untyped_defs mypy check for pyspark.pandas.base.

[ https://issues.apache.org/jira/browse/SPARK-35470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35470: Assignee: Apache Spark > Enable disallow_untyped_defs mypy check for pyspark.pandas.base.

[jira] [Updated] (SPARK-35652) Different Behaviour join vs joinWith in self joining

[ https://issues.apache.org/jira/browse/SPARK-35652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35652: -- Fix Version/s: (was: 3.0.3) 3.0.4 > Different Behaviour join vs joinWit

[jira] [Updated] (SPARK-35767) CoalesceExec can execute child plan twice

[ https://issues.apache.org/jira/browse/SPARK-35767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35767: -- Fix Version/s: (was: 3.0.3) 3.0.4 > CoalesceExec can execute child plan

[jira] [Updated] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35796: -- Fix Version/s: (was: 3.0.3) 3.0.4 > UT `handles k8s cluster mode` fails

[jira] [Updated] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35796: -- Issue Type: Bug (was: Improvement) > UT `handles k8s cluster mode` fails on MacOs >= 10.15 >

[jira] [Updated] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35796: -- Affects Version/s: (was: 3.1.3) 3.2.0 3.0.3

[jira] [Updated] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35796: -- Component/s: Spark Core > UT `handles k8s cluster mode` fails on MacOs >= 10.15 >

[jira] [Resolved] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35796. --- Fix Version/s: 3.0.3 3.1.3 3.2.0 Resolution: Fix

[jira] [Assigned] (SPARK-35796) UT `handles k8s cluster mode` fails on MacOs >= 10.15

[ https://issues.apache.org/jira/browse/SPARK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35796: - Assignee: Yazhi Wang > UT `handles k8s cluster mode` fails on MacOs >= 10.15 >

[jira] [Commented] (SPARK-35593) Support shuffle data recovery on the reused PVCs

[ https://issues.apache.org/jira/browse/SPARK-35593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365826#comment-17365826 ] Apache Spark commented on SPARK-35593: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35593) Support shuffle data recovery on the reused PVCs

[ https://issues.apache.org/jira/browse/SPARK-35593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365825#comment-17365825 ] Apache Spark commented on SPARK-35593: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35810) Remove ps.broadcast API

[ https://issues.apache.org/jira/browse/SPARK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365823#comment-17365823 ] Takuya Ueshin commented on SPARK-35810: --- I think we should deprecate it first and

[jira] [Commented] (SPARK-35808) Always enable the `pandas_metadata` in DataFrame.parquet

[ https://issues.apache.org/jira/browse/SPARK-35808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365822#comment-17365822 ] Kevin Su commented on SPARK-35808: -- [~ueshin] Thanks for the reply. Got it. > Always e

[jira] [Updated] (SPARK-35805) API auditing in Pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-35805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35805: - Target Version/s: 3.2.0 Priority: Blocker (was: Major) > API auditing in Pandas API

[jira] [Updated] (SPARK-35805) API auditing in Pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-35805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35805: - Summary: API auditing in Pandas API on Spark (was: Pandas API on Spark improvements) > API aud

[jira] [Resolved] (SPARK-35808) Always enable the `pandas_metadata` in DataFrame.parquet

[ https://issues.apache.org/jira/browse/SPARK-35808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-35808. --- Resolution: Won't Do I'd close this. > Always enable the `pandas_metadata` in DataFrame.par

[jira] [Resolved] (SPARK-35806) Rename the `mode` argument to avoid confusion with `mode` argument in pandas

[ https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-35806. - Resolution: Invalid Seems like we need more discussion. Let me just close this for now. > Renam

[jira] [Resolved] (SPARK-35807) Rename the `num_files` argument

[ https://issues.apache.org/jira/browse/SPARK-35807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-35807. - Resolution: Invalid Seems like we need more discussion. Let me just close this for now. > Renam

[jira] [Commented] (SPARK-35808) Always enable the `pandas_metadata` in DataFrame.parquet

[ https://issues.apache.org/jira/browse/SPARK-35808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365816#comment-17365816 ] Takuya Ueshin commented on SPARK-35808: --- I think we shouldn't do this. The reason

[jira] [Resolved] (SPARK-35565) Add a config for ignoring metadata directory of file stream sink

[ https://issues.apache.org/jira/browse/SPARK-35565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-35565. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32702 [https://gi

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365764#comment-17365764 ] Bruce Robbins commented on SPARK-35817: --- [~xkrogen] Thanks! {quote}I guess we shou

[jira] [Comment Edited] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365762#comment-17365762 ] Erik Krogen edited comment on SPARK-35817 at 6/18/21, 10:09 PM: --

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365762#comment-17365762 ] Erik Krogen commented on SPARK-35817: - Thanks for catching this [~bersprockets]! I w

[jira] [Updated] (SPARK-35818) Upgrade SBT to 1.5.4

[ https://issues.apache.org/jira/browse/SPARK-35818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35818: -- Parent: SPARK-35781 Issue Type: Sub-task (was: Improvement) > Upgrade SBT to 1.5.4 >

[jira] [Commented] (SPARK-35818) Upgrade SBT to 1.5.4

[ https://issues.apache.org/jira/browse/SPARK-35818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365750#comment-17365750 ] Apache Spark commented on SPARK-35818: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-35818) Upgrade SBT to 1.5.4

[ https://issues.apache.org/jira/browse/SPARK-35818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35818: Assignee: (was: Apache Spark) > Upgrade SBT to 1.5.4 > > >

[jira] [Assigned] (SPARK-35818) Upgrade SBT to 1.5.4

[ https://issues.apache.org/jira/browse/SPARK-35818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35818: Assignee: Apache Spark > Upgrade SBT to 1.5.4 > > >

[jira] [Commented] (SPARK-35818) Upgrade SBT to 1.5.4

[ https://issues.apache.org/jira/browse/SPARK-35818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365748#comment-17365748 ] Apache Spark commented on SPARK-35818: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-35818) Upgrade SBT to 1.5.4

Dongjoon Hyun created SPARK-35818: - Summary: Upgrade SBT to 1.5.4 Key: SPARK-35818 URL: https://issues.apache.org/jira/browse/SPARK-35818 Project: Spark Issue Type: Improvement Comp

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365697#comment-17365697 ] Bruce Robbins commented on SPARK-35817: --- The referenced line of code is meant to r

[jira] [Created] (SPARK-35817) Queries against wide Avro tables can be slow

Bruce Robbins created SPARK-35817: - Summary: Queries against wide Avro tables can be slow Key: SPARK-35817 URL: https://issues.apache.org/jira/browse/SPARK-35817 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-35478) Enable disallow_untyped_defs mypy check for pyspark.pandas.window.

[ https://issues.apache.org/jira/browse/SPARK-35478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365696#comment-17365696 ] Apache Spark commented on SPARK-35478: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-35478) Enable disallow_untyped_defs mypy check for pyspark.pandas.window.

[ https://issues.apache.org/jira/browse/SPARK-35478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365695#comment-17365695 ] Apache Spark commented on SPARK-35478: -- User 'ueshin' has created a pull request fo

[jira] [Resolved] (SPARK-35478) Enable disallow_untyped_defs mypy check for pyspark.pandas.window.

[ https://issues.apache.org/jira/browse/SPARK-35478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-35478. --- Fix Version/s: 3.2.0 Assignee: Kevin Su Resolution: Fixed Issue resolved by

[jira] [Resolved] (SPARK-35342) Introduce DecimalOps

[ https://issues.apache.org/jira/browse/SPARK-35342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-35342. --- Fix Version/s: 3.2.0 Assignee: Yikun Jiang Resolution: Fixed Issue resolved

[jira] [Updated] (SPARK-35816) Spark read write with multiple Hadoop HA cluster limitation

[ https://issues.apache.org/jira/browse/SPARK-35816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Jain updated SPARK-35816: Description: I have two Hadoop HA cluster: h1 and h2. Want to read from h1-HDFS and write to h2-H

[jira] [Created] (SPARK-35816) Spark read write with multiple Hadoop HA cluster limitation

Anupam Jain created SPARK-35816: --- Summary: Spark read write with multiple Hadoop HA cluster limitation Key: SPARK-35816 URL: https://issues.apache.org/jira/browse/SPARK-35816 Project: Spark Is

[jira] [Comment Edited] (SPARK-35756) unionByName should support nested struct also

[ https://issues.apache.org/jira/browse/SPARK-35756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365579#comment-17365579 ] Wassim Almaaoui edited comment on SPARK-35756 at 6/18/21, 4:26 PM: ---

[jira] [Commented] (SPARK-35756) unionByName should support nested struct also

[ https://issues.apache.org/jira/browse/SPARK-35756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365579#comment-17365579 ] Wassim Almaaoui commented on SPARK-35756: - I was not expecting this to work, we

[jira] [Commented] (SPARK-35756) unionByName should support nested struct also

[ https://issues.apache.org/jira/browse/SPARK-35756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365549#comment-17365549 ] Saurabh Chawla commented on SPARK-35756: This will work struct also  if allowMis

[jira] [Commented] (SPARK-35815) Allow delayThreshold for watermark to be represented as ANSI day-time/year-month interval literals

[ https://issues.apache.org/jira/browse/SPARK-35815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365547#comment-17365547 ] Kousuke Saruta commented on SPARK-35815: Waiting for SPARK-35749 and SPARK-35773

[jira] [Created] (SPARK-35815) Allow delayThreshold for watermark to be represented as ANSI day-time/year-month interval literals

Kousuke Saruta created SPARK-35815: -- Summary: Allow delayThreshold for watermark to be represented as ANSI day-time/year-month interval literals Key: SPARK-35815 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Commented] (SPARK-35811) Deprecate DataFrame.to_spark_io

[ https://issues.apache.org/jira/browse/SPARK-35811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365517#comment-17365517 ] Apache Spark commented on SPARK-35811: -- User 'pingsutw' has created a pull request

[jira] [Assigned] (SPARK-35811) Deprecate DataFrame.to_spark_io

[ https://issues.apache.org/jira/browse/SPARK-35811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35811: Assignee: Apache Spark > Deprecate DataFrame.to_spark_io > --

[jira] [Assigned] (SPARK-35811) Deprecate DataFrame.to_spark_io

[ https://issues.apache.org/jira/browse/SPARK-35811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35811: Assignee: (was: Apache Spark) > Deprecate DataFrame.to_spark_io > ---

[jira] [Assigned] (SPARK-34120) Improve the statistics estimation

[ https://issues.apache.org/jira/browse/SPARK-34120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-34120: --- Assignee: Yuming Wang > Improve the statistics estimation > ---

[jira] [Resolved] (SPARK-34120) Improve the statistics estimation

[ https://issues.apache.org/jira/browse/SPARK-34120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-34120. - Fix Version/s: 3.2.0 Resolution: Fixed > Improve the statistics estimation >

[jira] [Assigned] (SPARK-35185) Improve Distinct statistics estimation

[ https://issues.apache.org/jira/browse/SPARK-35185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-35185: --- Assignee: Yuming Wang > Improve Distinct statistics estimation > --

[jira] [Resolved] (SPARK-35185) Improve Distinct statistics estimation

[ https://issues.apache.org/jira/browse/SPARK-35185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-35185. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32291 [https://gith

[jira] [Commented] (SPARK-35808) Always enable the `pandas_metadata` in DataFrame.parquet

[ https://issues.apache.org/jira/browse/SPARK-35808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365506#comment-17365506 ] Kevin Su commented on SPARK-35808: -- [~itholic] So could we remove argument "pandas_meta

[jira] [Comment Edited] (SPARK-35700) spark.sql.orc.filterPushdown not working with Spark 3.1.1 for tables with varchar data type

[ https://issues.apache.org/jira/browse/SPARK-35700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365492#comment-17365492 ] Saurabh Chawla edited comment on SPARK-35700 at 6/18/21, 1:10 PM:

[jira] [Commented] (SPARK-35795) Cannot resolve column when there is a unrecognized hint in subquery

[ https://issues.apache.org/jira/browse/SPARK-35795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365495#comment-17365495 ] HonglunChen commented on SPARK-35795: - [~saurabhc100] Thanks > Cannot resolve colum

[jira] [Commented] (SPARK-35700) spark.sql.orc.filterPushdown not working with Spark 3.1.1 for tables with varchar data type

[ https://issues.apache.org/jira/browse/SPARK-35700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365492#comment-17365492 ] Saurabh Chawla commented on SPARK-35700: [~arghya18] - I tried to reproduce this

[jira] [Comment Edited] (SPARK-35089) non consistent results running count for same dataset after filter and lead window function

[ https://issues.apache.org/jira/browse/SPARK-35089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365468#comment-17365468 ] Domagoj edited comment on SPARK-35089 at 6/18/21, 12:44 PM:

[jira] [Updated] (SPARK-35787) Does anyone has performance issue after upgrade from 3.0 to 3.1?

[ https://issues.apache.org/jira/browse/SPARK-35787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vidmantas Drasutis updated SPARK-35787: --- Description: Hello.   We had using spark 3.0.2 and query was executed in ~100 seco

[jira] [Assigned] (SPARK-35608) Support AQE optimizer side transformUpWithPruning

[ https://issues.apache.org/jira/browse/SPARK-35608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-35608: -- Assignee: XiDuo You > Support AQE optimizer side transformUpWithPruning > ---

[jira] [Resolved] (SPARK-35608) Support AQE optimizer side transformUpWithPruning

[ https://issues.apache.org/jira/browse/SPARK-35608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-35608. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32742 [https:

[jira] [Commented] (SPARK-35795) Cannot resolve column when there is a unrecognized hint in subquery

[ https://issues.apache.org/jira/browse/SPARK-35795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365471#comment-17365471 ] Saurabh Chawla commented on SPARK-35795: This is fixed as the part of the - SPAR

[jira] [Resolved] (SPARK-35469) Enable disallow_untyped_defs mypy check for pyspark.pandas.accessors.

[ https://issues.apache.org/jira/browse/SPARK-35469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35469. -- Fix Version/s: 3.2.0 Assignee: Takuya Ueshin Resolution: Fixed > Enable disall

[jira] [Commented] (SPARK-35089) non consistent results running count for same dataset after filter and lead window function

[ https://issues.apache.org/jira/browse/SPARK-35089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365468#comment-17365468 ] Domagoj commented on SPARK-35089: - [~revans2], tnx for detailed explanation. I still ha

[jira] [Assigned] (SPARK-35678) add a common softmax function

[ https://issues.apache.org/jira/browse/SPARK-35678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-35678: Assignee: zhengruifeng > add a common softmax function > - >

[jira] [Assigned] (SPARK-35619) Refactor LinearRegression - make huber support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-35619: Assignee: zhengruifeng > Refactor LinearRegression - make huber support virtual centering

[jira] [Assigned] (SPARK-35100) Refactor AFT - support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-35100: Assignee: zhengruifeng > Refactor AFT - support virtual centering > -

[jira] [Assigned] (SPARK-35024) Refactor LinearSVC - support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-35024: Assignee: zhengruifeng > Refactor LinearSVC - support virtual centering > ---

[jira] [Updated] (SPARK-35024) Refactor LinearSVC - support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-35024: - Fix Version/s: 3.2.0 > Refactor LinearSVC - support virtual centering >

[jira] [Updated] (SPARK-35619) Refactor LinearRegression - make huber support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-35619: - Fix Version/s: 3.2.0 > Refactor LinearRegression - make huber support virtual centering > --

[jira] [Updated] (SPARK-35100) Refactor AFT - support virtual centering

[ https://issues.apache.org/jira/browse/SPARK-35100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-35100: - Fix Version/s: 3.2.0 > Refactor AFT - support virtual centering > --

[jira] [Updated] (SPARK-35666) add new gemv to skip array shape checking

[ https://issues.apache.org/jira/browse/SPARK-35666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-35666: - Fix Version/s: 3.2.0 > add new gemv to skip array shape checking > -

[jira] [Updated] (SPARK-35707) optimize sparse GEMM by skipping bound checking

[ https://issues.apache.org/jira/browse/SPARK-35707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-35707: - Fix Version/s: 3.2.0 > optimize sparse GEMM by skipping bound checking > ---

[jira] [Commented] (SPARK-35678) add a common softmax function

[ https://issues.apache.org/jira/browse/SPARK-35678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365454#comment-17365454 ] zhengruifeng commented on SPARK-35678: -- [~hyukjin.kwon] Thanks for reminding me. I

[jira] [Created] (SPARK-35814) Mismatched types when creating a new column when using Arrow

Nic Crane created SPARK-35814: - Summary: Mismatched types when creating a new column when using Arrow Key: SPARK-35814 URL: https://issues.apache.org/jira/browse/SPARK-35814 Project: Spark Issue

[jira] [Commented] (SPARK-35378) Eagerly execute commands in QueryExecution instead of caller sides

[ https://issues.apache.org/jira/browse/SPARK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365415#comment-17365415 ] Apache Spark commented on SPARK-35378: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-35378) Eagerly execute commands in QueryExecution instead of caller sides

[ https://issues.apache.org/jira/browse/SPARK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365413#comment-17365413 ] Apache Spark commented on SPARK-35378: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-35769) Truncate java.time.Period by fields of year-month interval type

[ https://issues.apache.org/jira/browse/SPARK-35769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-35769: Assignee: angerszhu > Truncate java.time.Period by fields of year-month interval type > -

[jira] [Resolved] (SPARK-35769) Truncate java.time.Period by fields of year-month interval type

[ https://issues.apache.org/jira/browse/SPARK-35769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-35769. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32945 [https://github.com

[jira] [Commented] (SPARK-35498) Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode

[ https://issues.apache.org/jira/browse/SPARK-35498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365310#comment-17365310 ] Apache Spark commented on SPARK-35498: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-35498) Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode

[ https://issues.apache.org/jira/browse/SPARK-35498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365309#comment-17365309 ] Apache Spark commented on SPARK-35498: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-35303) Enable pinned thread mode by default

[ https://issues.apache.org/jira/browse/SPARK-35303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365307#comment-17365307 ] Apache Spark commented on SPARK-35303: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-35498) Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode

[ https://issues.apache.org/jira/browse/SPARK-35498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365308#comment-17365308 ] Apache Spark commented on SPARK-35498: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-35303) Enable pinned thread mode by default

[ https://issues.apache.org/jira/browse/SPARK-35303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365305#comment-17365305 ] Apache Spark commented on SPARK-35303: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-35678) add a common softmax function

[ https://issues.apache.org/jira/browse/SPARK-35678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365294#comment-17365294 ] Hyukjin Kwon commented on SPARK-35678: -- [~podongfeng] please set the fix version ..

[jira] [Updated] (SPARK-35678) add a common softmax function

[ https://issues.apache.org/jira/browse/SPARK-35678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35678: - Fix Version/s: 3.2.0 > add a common softmax function > - > >

[jira] [Commented] (SPARK-35694) Increase the default JVM stack size of SBT/Maven

[ https://issues.apache.org/jira/browse/SPARK-35694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365288#comment-17365288 ] Apache Spark commented on SPARK-35694: -- User 'gengliangwang' has created a pull req

[jira] [Commented] (SPARK-35694) Increase the default JVM stack size of SBT/Maven

[ https://issues.apache.org/jira/browse/SPARK-35694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365287#comment-17365287 ] Apache Spark commented on SPARK-35694: -- User 'gengliangwang' has created a pull req