[jira] [Resolved] (SPARK-37137) Inline type hints for python/pyspark/conf.py

[ https://issues.apache.org/jira/browse/SPARK-37137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37137. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34411

[jira] [Assigned] (SPARK-37137) Inline type hints for python/pyspark/conf.py

[ https://issues.apache.org/jira/browse/SPARK-37137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37137: Assignee: Byron Hsu > Inline type hints for python/pyspark/conf.py >

[jira] [Assigned] (SPARK-37211) More descriptions and adding an image to the failure message about enabling GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-37211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37211: Assignee: (was: Apache Spark) > More descriptions and adding an image to the failure

[jira] [Assigned] (SPARK-37211) More descriptions and adding an image to the failure message about enabling GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-37211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37211: Assignee: Apache Spark > More descriptions and adding an image to the failure message

[jira] [Commented] (SPARK-37211) More descriptions and adding an image to the failure message about enabling GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-37211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17439042#comment-17439042 ] Apache Spark commented on SPARK-37211: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37157) Inline type hints for python/pyspark/util.py

[ https://issues.apache.org/jira/browse/SPARK-37157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37157. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34438

[jira] [Assigned] (SPARK-37157) Inline type hints for python/pyspark/util.py

[ https://issues.apache.org/jira/browse/SPARK-37157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37157: Assignee: dch nguyen > Inline type hints for python/pyspark/util.py >

[jira] [Updated] (SPARK-37210) An error occurred while concurrently writing to different static partitions

[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Wang updated SPARK-37210: -- Description: An error occurred while concurrently writing to different static partitions. For

[jira] [Created] (SPARK-37211) More descriptions and adding an image to the failure message about enabling GitHub Actions

Yuto Akutsu created SPARK-37211: --- Summary: More descriptions and adding an image to the failure message about enabling GitHub Actions Key: SPARK-37211 URL: https://issues.apache.org/jira/browse/SPARK-37211

[jira] [Commented] (SPARK-37210) An error occurred while concurrently writing to different static partitions

[ https://issues.apache.org/jira/browse/SPARK-37210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17439025#comment-17439025 ] Zhen Wang commented on SPARK-37210: --- The test code can be executed normally in spark 2.4.3. I noticed

[jira] [Created] (SPARK-37210) An error occurred while concurrently writing to different static partitions

Zhen Wang created SPARK-37210: - Summary: An error occurred while concurrently writing to different static partitions Key: SPARK-37210 URL: https://issues.apache.org/jira/browse/SPARK-37210 Project: Spark

[jira] [Commented] (SPARK-35496) Upgrade Scala 2.13 to 2.13.7

[ https://issues.apache.org/jira/browse/SPARK-35496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438986#comment-17438986 ] Kousuke Saruta commented on SPARK-35496: [~dongjoon] Thank you for letting me know. That's

[jira] [Commented] (SPARK-35496) Upgrade Scala 2.13 to 2.13.7

[ https://issues.apache.org/jira/browse/SPARK-35496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438984#comment-17438984 ] Dongjoon Hyun commented on SPARK-35496: --- Hi, [~sarutak]. [~LuciferYang] already updated his PR.

[jira] [Commented] (SPARK-36895) Add Create Index syntax support

[ https://issues.apache.org/jira/browse/SPARK-36895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438920#comment-17438920 ] Apache Spark commented on SPARK-36895: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Commented] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438880#comment-17438880 ] Apache Spark commented on SPARK-37208: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37208: Assignee: Apache Spark > Support mapping Spark gpu/fpga resource types to custom YARN

[jira] [Assigned] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37208: Assignee: (was: Apache Spark) > Support mapping Spark gpu/fpga resource types to

[jira] [Comment Edited] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-31726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438862#comment-17438862 ] Martin Andersson edited comment on SPARK-31726 at 11/4/21, 6:10 PM:

[jira] [Commented] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-31726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438862#comment-17438862 ] Martin Andersson commented on SPARK-31726: -- I've also run into this issue. Trying to simply

[jira] [Updated] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37209: - Attachment: success-unit-tests.log > YarnShuffleIntegrationSuite and other two similar cases in >

[jira] [Updated] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37209: - Attachment: failed-unit-tests.log > YarnShuffleIntegrationSuite and other two similar cases in >

[jira] [Updated] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

[ https://issues.apache.org/jira/browse/SPARK-37209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37209: - Description: Execute : # build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud

[jira] [Created] (SPARK-37209) YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed

Yang Jie created SPARK-37209: Summary: YarnShuffleIntegrationSuite and other two similar cases in `resource-managers` test failed Key: SPARK-37209 URL: https://issues.apache.org/jira/browse/SPARK-37209

[jira] [Comment Edited] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

[ https://issues.apache.org/jira/browse/SPARK-37198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438789#comment-17438789 ] Chuck Connell edited comment on SPARK-37198 at 11/4/21, 3:42 PM: - There

[jira] [Commented] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

[ https://issues.apache.org/jira/browse/SPARK-37198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438789#comment-17438789 ] Chuck Connell commented on SPARK-37198: --- There are many hints/techtips on the Internet which say

[jira] [Updated] (SPARK-37203) Fix NotSerializableException when observe with TypedImperativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-37203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-37203: --- Summary: Fix NotSerializableException when observe with TypedImperativeAggregate (was: Fix

[jira] [Commented] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438784#comment-17438784 ] Thomas Graves commented on SPARK-37208: --- Note, I'm working on this. > Support mapping Spark

[jira] [Created] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

Thomas Graves created SPARK-37208: - Summary: Support mapping Spark gpu/fpga resource types to custom YARN resource type Key: SPARK-37208 URL: https://issues.apache.org/jira/browse/SPARK-37208

[jira] [Assigned] (SPARK-37038) Sample push down in DS v2

[ https://issues.apache.org/jira/browse/SPARK-37038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37038: --- Assignee: Huaxin Gao > Sample push down in DS v2 > - > >

[jira] [Resolved] (SPARK-37038) Sample push down in DS v2

[ https://issues.apache.org/jira/browse/SPARK-37038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37038. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34451

[jira] [Commented] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438611#comment-17438611 ] Apache Spark commented on SPARK-37207: -- User 'dhirennavani' has created a pull request for this

[jira] [Commented] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438610#comment-17438610 ] Apache Spark commented on SPARK-37207: -- User 'dhirennavani' has created a pull request for this

[jira] [Commented] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438563#comment-17438563 ] Apache Spark commented on SPARK-37207: -- User 'dhirennavani' has created a pull request for this

[jira] [Commented] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438562#comment-17438562 ] Apache Spark commented on SPARK-37207: -- User 'dhirennavani' has created a pull request for this

[jira] [Assigned] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37207: Assignee: (was: Apache Spark) > Python API does not have isEmpty >

[jira] [Assigned] (SPARK-37207) Python API does not have isEmpty

[ https://issues.apache.org/jira/browse/SPARK-37207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37207: Assignee: Apache Spark > Python API does not have isEmpty >

[jira] [Created] (SPARK-37207) Python API does not have isEmpty

Dhiren Navani created SPARK-37207: - Summary: Python API does not have isEmpty Key: SPARK-37207 URL: https://issues.apache.org/jira/browse/SPARK-37207 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-35149) I am facing this issue regularly, how to fix this issue.

[ https://issues.apache.org/jira/browse/SPARK-35149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438538#comment-17438538 ] freedom1993 feng commented on SPARK-35149: -- how to solve this problem > I am facing this issue

[jira] [Commented] (SPARK-35496) Upgrade Scala 2.13 to 2.13.7

[ https://issues.apache.org/jira/browse/SPARK-35496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438535#comment-17438535 ] Kousuke Saruta commented on SPARK-35496: [~LuciferYang] Scala 2.13.7 was released a few days

[jira] [Resolved] (SPARK-37180) PySpark.pandas should support __version__

[ https://issues.apache.org/jira/browse/SPARK-37180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37180. -- Resolution: Won't Fix > PySpark.pandas should support __version__ >

[jira] [Commented] (SPARK-37180) PySpark.pandas should support __version__

[ https://issues.apache.org/jira/browse/SPARK-37180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438526#comment-17438526 ] Hyukjin Kwon commented on SPARK-37180: -- Yeah I think we don't need this. > PySpark.pandas should

[jira] [Commented] (SPARK-37180) PySpark.pandas should support __version__

[ https://issues.apache.org/jira/browse/SPARK-37180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438522#comment-17438522 ] dch nguyen commented on SPARK-37180: As Koalas was merged into Pyspark, so Should

[jira] [Comment Edited] (SPARK-37180) PySpark.pandas should support __version__

[ https://issues.apache.org/jira/browse/SPARK-37180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438522#comment-17438522 ] dch nguyen edited comment on SPARK-37180 at 11/4/21, 7:31 AM: -- As Koalas

[jira] [Updated] (SPARK-37201) Spark SQL reads unnecessary nested fields (filter after explode)

[ https://issues.apache.org/jira/browse/SPARK-37201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37201: - Issue Type: Improvement (was: Bug) > Spark SQL reads unnecessary nested fields (filter after

[jira] [Commented] (SPARK-37197) Behaviour inconsistency between pandas and pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-37197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438504#comment-17438504 ] Hyukjin Kwon commented on SPARK-37197: -- cc [~yikunkero] [~ueshin] [~itholic] [~XinrongM] [~dchvn]

[jira] [Updated] (SPARK-37197) Behaviour inconsistency between pandas and pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-37197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37197: - Description: This JIRA includes tickets on inconsistent behaviors(was: SPARK-37180  

[jira] [Updated] (SPARK-37197) Behaviour inconsistency between pandas and pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-37197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37197: - Description: This JIRA includes tickets on inconsistent behaviors pandas and pandas API on

[jira] [Resolved] (SPARK-37206) Upgrade Avro to 1.11.0

[ https://issues.apache.org/jira/browse/SPARK-37206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37206. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34482

[jira] [Updated] (SPARK-37189) pyspark.pandas histogram accepts the range option but does not use it

[ https://issues.apache.org/jira/browse/SPARK-37189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37189: - Parent: SPARK-37197 Issue Type: Sub-task (was: Bug) > pyspark.pandas histogram accepts

[jira] [Updated] (SPARK-37198) pyspark.pandas read_csv() and to_csv() should handle local files

[ https://issues.apache.org/jira/browse/SPARK-37198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37198: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) > pyspark.pandas

[jira] [Updated] (SPARK-37197) Behaviour inconsistency between pandas and pandas API on Spark

[ https://issues.apache.org/jira/browse/SPARK-37197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37197: - Summary: Behaviour inconsistency between pandas and pandas API on Spark (was: PySpark pandas

[jira] [Updated] (SPARK-37184) pyspark.pandas should support DF["column"].str.split("some_suffix").str[0]

[ https://issues.apache.org/jira/browse/SPARK-37184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37184: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) > pyspark.pandas should

[jira] [Updated] (SPARK-37187) pyspark.pandas fails to create a histogram of one column from a large DataFrame

[ https://issues.apache.org/jira/browse/SPARK-37187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37187: - Parent: SPARK-37197 Issue Type: Sub-task (was: Bug) > pyspark.pandas fails to create a

[jira] [Updated] (SPARK-37183) pyspark.pandas.DataFrame.map() should support .fillna()

[ https://issues.apache.org/jira/browse/SPARK-37183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37183: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) >

[jira] [Updated] (SPARK-37186) pyspark.pandas should support tseries.offsets

[ https://issues.apache.org/jira/browse/SPARK-37186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37186: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) > pyspark.pandas should

[jira] [Updated] (SPARK-37188) pyspark.pandas histogram accepts the title option but does not add a title to the plot

[ https://issues.apache.org/jira/browse/SPARK-37188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37188: - Parent: SPARK-37197 Issue Type: Sub-task (was: Bug) > pyspark.pandas histogram accepts

[jira] [Updated] (SPARK-37181) pyspark.pandas.read_csv() should support latin-1 encoding

[ https://issues.apache.org/jira/browse/SPARK-37181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37181: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) >

[jira] [Updated] (SPARK-37180) PySpark.pandas should support __version__

[ https://issues.apache.org/jira/browse/SPARK-37180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37180: - Parent: SPARK-37197 Issue Type: Sub-task (was: Improvement) > PySpark.pandas should

[jira] [Resolved] (SPARK-37190) Improve error messages for casting under ANSI mode

[ https://issues.apache.org/jira/browse/SPARK-37190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37190. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34463

[jira] [Assigned] (SPARK-37190) Improve error messages for casting under ANSI mode

[ https://issues.apache.org/jira/browse/SPARK-37190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37190: --- Assignee: Allison Wang > Improve error messages for casting under ANSI mode >