[jira] [Updated] (SPARK-33386) Accessing array elements should failed if index is out of bound.

[ https://issues.apache.org/jira/browse/SPARK-33386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leanken.Lin updated SPARK-33386: Description: When ansi mode enabled, accessing array element with out of bound index should failed

[jira] [Updated] (SPARK-33460) Accessing map values should fail if key is not found.

[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leanken.Lin updated SPARK-33460: Description: When ansi mode enabled, accessing map values should failed with exception if key does

[jira] [Created] (SPARK-33460) Accessing map values should fail if key is not found.

Leanken.Lin created SPARK-33460: --- Summary: Accessing map values should fail if key is not found. Key: SPARK-33460 URL: https://issues.apache.org/jira/browse/SPARK-33460 Project: Spark Issue Typ

[jira] [Resolved] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33458. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30383 [https://gith

[jira] [Commented] (SPARK-33449) Add cache for Parquet Metadata

[ https://issues.apache.org/jira/browse/SPARK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232578#comment-17232578 ] Yuming Wang commented on SPARK-33449: - Thank you [~LuciferYang]. > Add cache for Pa

[jira] [Assigned] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33458: --- Assignee: Yuming Wang > Hive partition pruning support Contains, StartsWith and EndsWith pr

[jira] [Assigned] (SPARK-33394) Throw `NoSuchDatabaseException` for not existing namespace in DSv2 SHOW TABLES

[ https://issues.apache.org/jira/browse/SPARK-33394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33394: --- Assignee: Maxim Gekk > Throw `NoSuchDatabaseException` for not existing namespace in DSv2 S

[jira] [Resolved] (SPARK-33394) Throw `NoSuchDatabaseException` for not existing namespace in DSv2 SHOW TABLES

[ https://issues.apache.org/jira/browse/SPARK-33394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33394. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30358 [https://gith

[jira] [Commented] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232573#comment-17232573 ] Apache Spark commented on SPARK-33456: -- User 'viirya' has created a pull request fo

[jira] [Resolved] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33456. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30381 [https://gi

[jira] [Updated] (SPARK-32726) Filter by column alias in where clause

[ https://issues.apache.org/jira/browse/SPARK-32726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32726: Parent: SPARK-33459 Issue Type: Sub-task (was: Improvement) > Filter by column alias in w

[jira] [Updated] (SPARK-33245) Add built-in UDF - GETBIT

[ https://issues.apache.org/jira/browse/SPARK-33245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33245: Parent: SPARK-33459 Issue Type: Sub-task (was: Improvement) > Add built-in UDF - GETBIT

[jira] [Updated] (SPARK-31561) Add QUALIFY Clause

[ https://issues.apache.org/jira/browse/SPARK-31561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31561: Parent: SPARK-33459 Issue Type: Sub-task (was: Improvement) > Add QUALIFY Clause > --

[jira] [Updated] (SPARK-31634) "show tables like" support for SQL wildcard characters (% and _)

[ https://issues.apache.org/jira/browse/SPARK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31634: Parent: SPARK-33459 Issue Type: Sub-task (was: Improvement) > "show tables like" support

[jira] [Commented] (SPARK-33183) Bug in optimizer rule EliminateSorts

[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232560#comment-17232560 ] Dongjoon Hyun commented on SPARK-33183: --- Hi, [~allisonwang-db]. Does Apache Spark

[jira] [Assigned] (SPARK-33454) Add GitHub Action job for Hadoop 2

[ https://issues.apache.org/jira/browse/SPARK-33454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33454: Assignee: Dongjoon Hyun > Add GitHub Action job for Hadoop 2 > --

[jira] [Resolved] (SPARK-33454) Add GitHub Action job for Hadoop 2

[ https://issues.apache.org/jira/browse/SPARK-33454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33454. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30378 [https://gi

[jira] [Updated] (SPARK-30724) Support 'like any' and 'like all' operators

[ https://issues.apache.org/jira/browse/SPARK-30724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-30724: Parent: SPARK-33459 Issue Type: Sub-task (was: New Feature) > Support 'like any' and 'lik

[jira] [Updated] (SPARK-33410) Resolve SQL query reference a column by an alias

[ https://issues.apache.org/jira/browse/SPARK-33410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-33410: Parent: SPARK-33459 Issue Type: Sub-task (was: Improvement) > Resolve SQL query reference

[jira] [Created] (SPARK-33459) Commonly used Teradata extension syntax

Yuming Wang created SPARK-33459: --- Summary: Commonly used Teradata extension syntax Key: SPARK-33459 URL: https://issues.apache.org/jira/browse/SPARK-33459 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-33260) SortExec produces incorrect results if sortOrder is a Stream

[ https://issues.apache.org/jira/browse/SPARK-33260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33260: -- Affects Version/s: 3.1.0 > SortExec produces incorrect results if sortOrder is a Stream >

[jira] [Updated] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33358: -- Affects Version/s: 3.1.0 > Spark SQL CLI command processing loop can't exit while one comand f

[jira] [Updated] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

[ https://issues.apache.org/jira/browse/SPARK-33290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33290: -- Affects Version/s: 3.1.0 > REFRESH TABLE should invalidate cache even though the table itself

[jira] [Commented] (SPARK-33445) Can't parse decimal type from csv file

[ https://issues.apache.org/jira/browse/SPARK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232546#comment-17232546 ] Yang Jie commented on SPARK-33445: -- It seems the master branch  infer the schema as Dou

[jira] [Comment Edited] (SPARK-33449) Add cache for Parquet Metadata

[ https://issues.apache.org/jira/browse/SPARK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232478#comment-17232478 ] Yang Jie edited comment on SPARK-33449 at 11/16/20, 2:59 AM: -

[jira] [Commented] (SPARK-33449) Add cache for Parquet Metadata

[ https://issues.apache.org/jira/browse/SPARK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232478#comment-17232478 ] Yang Jie commented on SPARK-33449: -- [~yumwang] Yes, I think this is a very good suggest

[jira] [Updated] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33358: -- Priority: Critical (was: Major) > Spark SQL CLI command processing loop can't exit while one

[jira] [Commented] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232464#comment-17232464 ] Apache Spark commented on SPARK-33458: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33458: Assignee: Apache Spark > Hive partition pruning support Contains, StartsWith and EndsWith

[jira] [Commented] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232463#comment-17232463 ] Apache Spark commented on SPARK-33458: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

[ https://issues.apache.org/jira/browse/SPARK-33458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33458: Assignee: (was: Apache Spark) > Hive partition pruning support Contains, StartsWith a

[jira] [Commented] (SPARK-33457) Adjust mypy configuration

[ https://issues.apache.org/jira/browse/SPARK-33457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232462#comment-17232462 ] Apache Spark commented on SPARK-33457: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-33457) Adjust mypy configuration

[ https://issues.apache.org/jira/browse/SPARK-33457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33457: Assignee: (was: Apache Spark) > Adjust mypy configuration > -

[jira] [Assigned] (SPARK-33457) Adjust mypy configuration

[ https://issues.apache.org/jira/browse/SPARK-33457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33457: Assignee: Apache Spark > Adjust mypy configuration > - > >

[jira] [Created] (SPARK-33458) Hive partition pruning support Contains, StartsWith and EndsWith predicate

Yuming Wang created SPARK-33458: --- Summary: Hive partition pruning support Contains, StartsWith and EndsWith predicate Key: SPARK-33458 URL: https://issues.apache.org/jira/browse/SPARK-33458 Project: Spa

[jira] [Created] (SPARK-33457) Adjust mypy configuration

Maciej Szymkiewicz created SPARK-33457: -- Summary: Adjust mypy configuration Key: SPARK-33457 URL: https://issues.apache.org/jira/browse/SPARK-33457 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-33253) Migration to NumPy documentation style in Streaming (pyspark.streaming.*)

[ https://issues.apache.org/jira/browse/SPARK-33253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33253: Assignee: Haejoon Lee > Migration to NumPy documentation style in Streaming (pyspark.stre

[jira] [Resolved] (SPARK-33253) Migration to NumPy documentation style in Streaming (pyspark.streaming.*)

[ https://issues.apache.org/jira/browse/SPARK-33253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33253. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30346 [https://gi

[jira] [Commented] (SPARK-33424) Doubts about the use of the "DiskBlockObjectWriter#revertPartialWritesAndClose" method in Spark Code

[ https://issues.apache.org/jira/browse/SPARK-33424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232451#comment-17232451 ] Yang Jie commented on SPARK-33424: -- OK [~hyukjin.kwon] > Doubts about the use of the

[jira] [Resolved] (SPARK-33451) change 'spark.sql.adaptive.skewedPartitionThresholdInBytes' to 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes'

[ https://issues.apache.org/jira/browse/SPARK-33451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33451. -- Fix Version/s: (was: 3.0.1) (was: 3.0.0) 3.0.2

[jira] [Assigned] (SPARK-33451) change 'spark.sql.adaptive.skewedPartitionThresholdInBytes' to 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes'

[ https://issues.apache.org/jira/browse/SPARK-33451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33451: Assignee: aof > change 'spark.sql.adaptive.skewedPartitionThresholdInBytes' to > 'spark.

[jira] [Resolved] (SPARK-33254) Migration to NumPy documentation style in Core (pyspark.*, pyspark.resource.*, etc.)

[ https://issues.apache.org/jira/browse/SPARK-33254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33254. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30320 [https://gi

[jira] [Assigned] (SPARK-33254) Migration to NumPy documentation style in Core (pyspark.*, pyspark.resource.*, etc.)

[ https://issues.apache.org/jira/browse/SPARK-33254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33254: Assignee: Maciej Szymkiewicz > Migration to NumPy documentation style in Core (pyspark.*,

[jira] [Assigned] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33358: - Assignee: Lichuanliang > Spark SQL CLI command processing loop can't exit while one com

[jira] [Resolved] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33358. --- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pul

[jira] [Commented] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232430#comment-17232430 ] Dongjoon Hyun commented on SPARK-33358: --- I verified this and updated the PR descri

[jira] [Updated] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33358: -- Affects Version/s: 3.0.0 > Spark SQL CLI command processing loop can't exit while one comand f

[jira] [Updated] (SPARK-33358) Spark SQL CLI command processing loop can't exit while one comand fail

[ https://issues.apache.org/jira/browse/SPARK-33358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33358: -- Labels: correctness (was: ) > Spark SQL CLI command processing loop can't exit while one coma

[jira] [Commented] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232417#comment-17232417 ] Apache Spark commented on SPARK-33456: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232416#comment-17232416 ] Apache Spark commented on SPARK-33456: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33456: Assignee: Apache Spark (was: L. C. Hsieh) > Add end-to-end test for subexpression elimin

[jira] [Assigned] (SPARK-33456) Add end-to-end test for subexpression elimination

[ https://issues.apache.org/jira/browse/SPARK-33456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33456: Assignee: L. C. Hsieh (was: Apache Spark) > Add end-to-end test for subexpression elimin

[jira] [Updated] (SPARK-32517) Add StorageLevel.DISK_ONLY_3

[ https://issues.apache.org/jira/browse/SPARK-32517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32517: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Improvement) > Add StorageLevel.DISK_O

[jira] [Updated] (SPARK-33218) Update misleading log messages for removed shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-33218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33218: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Bug) > Update misleading log messages

[jira] [Updated] (SPARK-33202) Fix BlockManagerDecommissioner to return the correct migration status

[ https://issues.apache.org/jira/browse/SPARK-33202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33202: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Bug) > Fix BlockManagerDecommissioner

[jira] [Updated] (SPARK-33387) Support ordered shuffle block migration

[ https://issues.apache.org/jira/browse/SPARK-33387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33387: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Improvement) > Support ordered shuffle

[jira] [Updated] (SPARK-33198) getMigrationBlocks should not fail at missing files

[ https://issues.apache.org/jira/browse/SPARK-33198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33198: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Bug) > getMigrationBlocks should not f

[jira] [Commented] (SPARK-33344) Fix Compilation warings of "multiarg infix syntax looks like a tuple and will be deprecated" in Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232414#comment-17232414 ] Guillaume Martres commented on SPARK-33344: --- Note that this warning will actua

[jira] [Created] (SPARK-33456) Add end-to-end test for subexpression elimination

L. C. Hsieh created SPARK-33456: --- Summary: Add end-to-end test for subexpression elimination Key: SPARK-33456 URL: https://issues.apache.org/jira/browse/SPARK-33456 Project: Spark Issue Type: T

[jira] [Commented] (SPARK-33295) Upgrade ORC to 1.6.6

[ https://issues.apache.org/jira/browse/SPARK-33295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232385#comment-17232385 ] Dongjoon Hyun commented on SPARK-33295: --- Oh, thank you for your correction by remo

[jira] [Assigned] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27421: Assignee: (was: Apache Spark) > RuntimeException when querying a view on a partitione

[jira] [Commented] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232312#comment-17232312 ] Apache Spark commented on SPARK-27421: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27421: Assignee: Apache Spark > RuntimeException when querying a view on a partitioned parquet t