[jira] [Commented] (SPARK-35320) from_json cannot parse maps with timestamp as key

[ https://issues.apache.org/jira/browse/SPARK-35320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387542#comment-17387542 ] Apache Spark commented on SPARK-35320: -- User 'planga82' has created a pull request for this issue:

[jira] [Commented] (SPARK-35320) from_json cannot parse maps with timestamp as key

[ https://issues.apache.org/jira/browse/SPARK-35320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387539#comment-17387539 ] Apache Spark commented on SPARK-35320: -- User 'planga82' has created a pull request for this issue:

[jira] [Resolved] (SPARK-32797) Install mypy on the Jenkins CI workers

[ https://issues.apache.org/jira/browse/SPARK-32797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-32797. - Resolution: Won't Fix re comments from [https://github.com/apache/spark/pull/33503] since we

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387452#comment-17387452 ] Apache Spark commented on SPARK-35259: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387454#comment-17387454 ] Apache Spark commented on SPARK-35259: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36290) Push down join condition evaluation

[ https://issues.apache.org/jira/browse/SPARK-36290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36290: Assignee: (was: Apache Spark) > Push down join condition evaluation >

[jira] [Assigned] (SPARK-36290) Push down join condition evaluation

[ https://issues.apache.org/jira/browse/SPARK-36290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36290: Assignee: Apache Spark > Push down join condition evaluation >

[jira] [Commented] (SPARK-36290) Push down join condition evaluation

[ https://issues.apache.org/jira/browse/SPARK-36290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387429#comment-17387429 ] Apache Spark commented on SPARK-36290: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-36290) Push down join condition evaluation

Yuming Wang created SPARK-36290: --- Summary: Push down join condition evaluation Key: SPARK-36290 URL: https://issues.apache.org/jira/browse/SPARK-36290 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Attachment: sample.csv > Issue with record count of data frame while reading in DropMalformed mode >

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Description: I am writing the steps to reproduce the issue for "count" pyspark api while using mode as

[jira] [Commented] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387404#comment-17387404 ] anju commented on SPARK-36277: -- [~hyukjin.kwon] I edited my issue description . Could you please check if

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Description: I am writing the steps to reproduce the issue for "count" pyspark api while using mode as

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Attachment: Inputfile.PNG > Issue with record count of data frame while reading in DropMalformed mode >

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Attachment: 111.PNG > Issue with record count of data frame while reading in DropMalformed mode >

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Attachment: (was: 111.PNG) > Issue with record count of data frame while reading in DropMalformed mode >

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Description: I am writing the steps to reproduce the issue for "count" pyspark api while using mode as

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Description: I am writing the steps to reproduce the issue for "count" pyspark api while using mode as

[jira] [Updated] (SPARK-36277) Issue with record count of data frame while reading in DropMalformed mode

[ https://issues.apache.org/jira/browse/SPARK-36277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anju updated SPARK-36277: - Description: I am writing the steps to reproduce the issue for "count" pyspark api while using mode as

[jira] [Assigned] (SPARK-36217) Rename CustomShuffleReader and OptimizeLocalShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-36217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36217: --- Assignee: Hyukjin Kwon > Rename CustomShuffleReader and OptimizeLocalShuffleReader >

[jira] [Resolved] (SPARK-36217) Rename CustomShuffleReader and OptimizeLocalShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-36217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36217. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33429

[jira] [Commented] (SPARK-36142) Adjust exponentiation between Series with missing values and bool literal to follow pandas

[ https://issues.apache.org/jira/browse/SPARK-36142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387346#comment-17387346 ] Apache Spark commented on SPARK-36142: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36142) Adjust exponentiation between Series with missing values and bool literal to follow pandas

[ https://issues.apache.org/jira/browse/SPARK-36142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36142: Assignee: Apache Spark > Adjust exponentiation between Series with missing values and

[jira] [Assigned] (SPARK-36142) Adjust exponentiation between Series with missing values and bool literal to follow pandas

[ https://issues.apache.org/jira/browse/SPARK-36142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36142: Assignee: (was: Apache Spark) > Adjust exponentiation between Series with missing

[jira] [Commented] (SPARK-35934) Extend SPARK-SQL to support REPEATABLE TABLESAMPLE clause.

[ https://issues.apache.org/jira/browse/SPARK-35934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387341#comment-17387341 ] Michail Giannakopoulos commented on SPARK-35934: [~hyukjin.kwon] [~sowen] Can I assign

[jira] [Assigned] (SPARK-36289) rewrite distinct count case when expressions without Expand node

[ https://issues.apache.org/jira/browse/SPARK-36289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36289: Assignee: (was: Apache Spark) > rewrite distinct count case when expressions without

[jira] [Commented] (SPARK-36289) rewrite distinct count case when expressions without Expand node

[ https://issues.apache.org/jira/browse/SPARK-36289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387336#comment-17387336 ] Apache Spark commented on SPARK-36289: -- User 'Gabriel39' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36289) rewrite distinct count case when expressions without Expand node

[ https://issues.apache.org/jira/browse/SPARK-36289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36289: Assignee: Apache Spark > rewrite distinct count case when expressions without Expand

[jira] [Created] (SPARK-36289) rewrite distinct count case when expressions without Expand node

Gabriel Lee created SPARK-36289: --- Summary: rewrite distinct count case when expressions without Expand node Key: SPARK-36289 URL: https://issues.apache.org/jira/browse/SPARK-36289 Project: Spark

[jira] [Commented] (SPARK-36034) Incorrect datetime filter when reading Parquet files written in legacy mode

[ https://issues.apache.org/jira/browse/SPARK-36034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387298#comment-17387298 ] Willi Raschkowski commented on SPARK-36034: --- [~maxgekk], much appreciated, thanks! >

[jira] [Assigned] (SPARK-36288) Update API usage on pyspark pandas documents

[ https://issues.apache.org/jira/browse/SPARK-36288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36288: Assignee: (was: Apache Spark) > Update API usage on pyspark pandas documents >

[jira] [Assigned] (SPARK-36288) Update API usage on pyspark pandas documents

[ https://issues.apache.org/jira/browse/SPARK-36288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36288: Assignee: Apache Spark > Update API usage on pyspark pandas documents >

[jira] [Commented] (SPARK-36288) Update API usage on pyspark pandas documents

[ https://issues.apache.org/jira/browse/SPARK-36288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387294#comment-17387294 ] Apache Spark commented on SPARK-36288: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Resolved] (SPARK-34402) Group exception about data format schema

[ https://issues.apache.org/jira/browse/SPARK-34402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34402. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33296

[jira] [Assigned] (SPARK-34402) Group exception about data format schema

[ https://issues.apache.org/jira/browse/SPARK-34402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34402: --- Assignee: angerszhu > Group exception about data format schema >

[jira] [Commented] (SPARK-34619) Update the Spark SQL guide about day-time and year-month interval types

[ https://issues.apache.org/jira/browse/SPARK-34619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387247#comment-17387247 ] Apache Spark commented on SPARK-34619: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34619) Update the Spark SQL guide about day-time and year-month interval types

[ https://issues.apache.org/jira/browse/SPARK-34619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34619: Assignee: Apache Spark > Update the Spark SQL guide about day-time and year-month

[jira] [Assigned] (SPARK-34619) Update the Spark SQL guide about day-time and year-month interval types

[ https://issues.apache.org/jira/browse/SPARK-34619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34619: Assignee: (was: Apache Spark) > Update the Spark SQL guide about day-time and

[jira] [Created] (SPARK-36288) Update API usage on pyspark pandas documents

Leona Yoda created SPARK-36288: -- Summary: Update API usage on pyspark pandas documents Key: SPARK-36288 URL: https://issues.apache.org/jira/browse/SPARK-36288 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36269) Fix only set data columns to Hive column names config

[ https://issues.apache.org/jira/browse/SPARK-36269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36269. - Fix Version/s: 3.1.3 3.2.0 3.0.4 Resolution: Fixed

[jira] [Assigned] (SPARK-36269) Fix only set data columns to Hive column names config

[ https://issues.apache.org/jira/browse/SPARK-36269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36269: --- Assignee: Cheng Su > Fix only set data columns to Hive column names config >

[jira] [Assigned] (SPARK-36105) OptimizeLocalShuffleReader support reading data of multiple mappers in one task

[ https://issues.apache.org/jira/browse/SPARK-36105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36105: --- Assignee: Michael Zhang > OptimizeLocalShuffleReader support reading data of multiple

[jira] [Resolved] (SPARK-36105) OptimizeLocalShuffleReader support reading data of multiple mappers in one task

[ https://issues.apache.org/jira/browse/SPARK-36105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36105. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33310

[jira] [Commented] (SPARK-36287) create TimestampNTZType in pyspark

[ https://issues.apache.org/jira/browse/SPARK-36287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387172#comment-17387172 ] Apache Spark commented on SPARK-36287: -- User 'dominikgehl' has created a pull request for this

[jira] [Assigned] (SPARK-36287) create TimestampNTZType in pyspark

[ https://issues.apache.org/jira/browse/SPARK-36287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36287: Assignee: Apache Spark > create TimestampNTZType in pyspark >

[jira] [Assigned] (SPARK-36287) create TimestampNTZType in pyspark

[ https://issues.apache.org/jira/browse/SPARK-36287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36287: Assignee: (was: Apache Spark) > create TimestampNTZType in pyspark >

[jira] [Assigned] (SPARK-34952) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-34952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34952: --- Assignee: (was: Apache Spark) > Aggregate (Min/Max/Count) push down for Parquet >

[jira] [Resolved] (SPARK-34952) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-34952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34952. - Fix Version/s: 3.2.0 Assignee: Huaxin Gao Resolution: Fixed > Aggregate

[jira] [Created] (SPARK-36287) create TimestampNTZType in pyspark

Dominik Gehl created SPARK-36287: Summary: create TimestampNTZType in pyspark Key: SPARK-36287 URL: https://issues.apache.org/jira/browse/SPARK-36287 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-34249) Add documentation for ANSI implicit cast rules

[ https://issues.apache.org/jira/browse/SPARK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387136#comment-17387136 ] Apache Spark commented on SPARK-34249: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-34249) Add documentation for ANSI implicit cast rules

[ https://issues.apache.org/jira/browse/SPARK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34249: Assignee: Apache Spark > Add documentation for ANSI implicit cast rules >

[jira] [Assigned] (SPARK-34249) Add documentation for ANSI implicit cast rules

[ https://issues.apache.org/jira/browse/SPARK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34249: Assignee: (was: Apache Spark) > Add documentation for ANSI implicit cast rules >

[jira] [Commented] (SPARK-34249) Add documentation for ANSI implicit cast rules

[ https://issues.apache.org/jira/browse/SPARK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387134#comment-17387134 ] Apache Spark commented on SPARK-34249: -- User 'gengliangwang' has created a pull request for this

[jira] [Updated] (SPARK-34249) Add documentation for ANSI implicit cast rules

[ https://issues.apache.org/jira/browse/SPARK-34249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-34249: --- Affects Version/s: (was: 3.1.2) 3.2.0 > Add documentation for

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- External issue URL: (was: https://issues.apache.org/jira/browse/SPARK-35211) > Bug when creating

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- External issue URL: https://issues.apache.org/jira/browse/SPARK-35211 > Bug when creating dataframe

[jira] [Assigned] (SPARK-36286) Block some invalid datetime string

[ https://issues.apache.org/jira/browse/SPARK-36286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36286: Assignee: (was: Apache Spark) > Block some invalid datetime string >

[jira] [Commented] (SPARK-36286) Block some invalid datetime string

[ https://issues.apache.org/jira/browse/SPARK-36286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387122#comment-17387122 ] Apache Spark commented on SPARK-36286: -- User 'linhongliu-db' has created a pull request for this

[jira] [Assigned] (SPARK-36286) Block some invalid datetime string

[ https://issues.apache.org/jira/browse/SPARK-36286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36286: Assignee: Apache Spark > Block some invalid datetime string >

[jira] [Created] (SPARK-36286) Block some invalid datetime string

Linhong Liu created SPARK-36286: --- Summary: Block some invalid datetime string Key: SPARK-36286 URL: https://issues.apache.org/jira/browse/SPARK-36286 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- Description: A reproducible small repo can be found here:

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- Description: h2. Case 1: Create PySpark Dataframe using Pandas DataFrame with Arrow disabled and

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- Description: h2. Case 1 Create PySpark Dataframe using Pandas DataFrame with Arrow disabled and

[jira] [Updated] (SPARK-36283) Bug when creating dataframe without schema and with Arrow disabled

[ https://issues.apache.org/jira/browse/SPARK-36283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Darcy Shen updated SPARK-36283: --- Description: h2. Case 1 {code:python} spark = SparkSession.builder.getOrCreate()

<    1   2