[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848787#comment-17848787 ] Bruce Robbins commented on SPARK-48361: --- Did you mean the following? {noformat} va

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849095#comment-17849095 ] Bruce Robbins commented on SPARK-48361: --- I can take a look at the root cause, unle

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849747#comment-17849747 ] Bruce Robbins commented on SPARK-48361: --- After looking at this, I see that this is

[jira] [Comment Edited] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849747#comment-17849747 ] Bruce Robbins edited comment on SPARK-48361 at 5/27/24 2:29 PM: --

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849785#comment-17849785 ] Bruce Robbins commented on SPARK-47193: --- Thanks for the update. This issue is see

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-06-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854211#comment-17854211 ] Bruce Robbins commented on SPARK-47193: --- I took a look at this today. This issue h

[jira] [Created] (SPARK-44477) CheckAnalysis uses error subclass as an error class

2023-07-18 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-44477: - Summary: CheckAnalysis uses error subclass as an error class Key: SPARK-44477 URL: https://issues.apache.org/jira/browse/SPARK-44477 Project: Spark Issue T

[jira] [Commented] (SPARK-44477) CheckAnalysis uses error subclass as an error class

2023-07-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744314#comment-17744314 ] Bruce Robbins commented on SPARK-44477: --- PR here: https://github.com/apache/spark/

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-08-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754344#comment-17754344 ] Bruce Robbins commented on SPARK-44805: --- It seems to be some weird interaction bet

[jira] [Comment Edited] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-08-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754344#comment-17754344 ] Bruce Robbins edited comment on SPARK-44805 at 8/15/23 12:26 AM: -

[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44805: -- Labels: correctness (was: ) > Data lost after union using > spark.sql.parquet.enableNestedCo

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-05 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762234#comment-17762234 ] Bruce Robbins commented on SPARK-44805: --- I looked at this yesterday and I think I

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762792#comment-17762792 ] Bruce Robbins commented on SPARK-44805: --- PR here: https://github.com/apache/spark/

[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44805: -- Affects Version/s: 3.4.1 > Data lost after union using > spark.sql.parquet.enableNestedColumn

[jira] [Created] (SPARK-45106) percentile_cont gets internal error when user input fails runtime replacement's input type check

2023-09-08 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45106: - Summary: percentile_cont gets internal error when user input fails runtime replacement's input type check Key: SPARK-45106 URL: https://issues.apache.org/jira/browse/SPARK-4510

[jira] [Updated] (SPARK-45106) percentile_cont gets internal error when user input fails runtime replacement's input type check

2023-09-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45106: -- Affects Version/s: 3.3.2 > percentile_cont gets internal error when user input fails runtime

[jira] [Commented] (SPARK-44912) Spark 3.4 multi-column sum slows with many columns

2023-09-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763455#comment-17763455 ] Bruce Robbins commented on SPARK-44912: --- It looks like this was fixed with SPARK-4

[jira] [Created] (SPARK-45171) GenerateExec fails to initialize non-deterministic expressions before use

2023-09-14 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45171: - Summary: GenerateExec fails to initialize non-deterministic expressions before use Key: SPARK-45171 URL: https://issues.apache.org/jira/browse/SPARK-45171 Project:

[jira] [Commented] (SPARK-45440) Incorrect summary counts from a CSV file

2023-10-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772724#comment-17772724 ] Bruce Robbins commented on SPARK-45440: --- I added {{inferSchema=true}} as a datasou

[jira] [Created] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45580: - Summary: RewritePredicateSubquery unexpectedly changes the output schema of certain queries Key: SPARK-45580 URL: https://issues.apache.org/jira/browse/SPARK-45580

[jira] [Updated] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Description: A query can have an incorrect output schema because of a subquery. Assume this d

[jira] [Commented] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776401#comment-17776401 ] Bruce Robbins commented on SPARK-45580: --- I'll make a PR in the coming days. > Rew

[jira] [Commented] (SPARK-45583) Spark SQL returning incorrect values for full outer join on keys with the same name.

2023-10-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776783#comment-17776783 ] Bruce Robbins commented on SPARK-45583: --- Strangely, I cannot reproduce. Is some se

[jira] [Commented] (SPARK-45601) stackoverflow when executing rule ExtractWindowExpressions

2023-10-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1304#comment-1304 ] Bruce Robbins commented on SPARK-45601: --- Possibly SPARK-38666 > stackoverflow whe

[jira] [Resolved] (SPARK-45583) Spark SQL returning incorrect values for full outer join on keys with the same name.

2023-10-20 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45583. --- Resolution: Fixed > Spark SQL returning incorrect values for full outer join on keys with th

[jira] [Updated] (SPARK-45580) Subquery changes the output schema of the outer query

2023-10-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Summary: Subquery changes the output schema of the outer query (was: Subquery changes the out

[jira] [Updated] (SPARK-45580) Subquery changes the output schema of outer query

2023-10-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Summary: Subquery changes the output schema of outer query (was: RewritePredicateSubquery une

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781091#comment-17781091 ] Bruce Robbins commented on SPARK-45644: --- You can turn on display of the generated

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781494#comment-17781494 ] Bruce Robbins commented on SPARK-45644: --- OK, I can reproduce. I will take a look.

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781531#comment-17781531 ] Bruce Robbins commented on SPARK-45644: --- I will look into it and try to submit a f

[jira] [Commented] (SPARK-45797) Discrepancies in PySpark DataFrame Results When Using Window Functions and Filters

2023-11-05 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783015#comment-17783015 ] Bruce Robbins commented on SPARK-45797: --- I wonder if this is the same as SPARK-455

[jira] [Created] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq]

2023-11-11 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45896: - Summary: Expression encoding fails for Seq/Map of Option[Seq] Key: SPARK-45896 URL: https://issues.apache.org/jira/browse/SPARK-45896 Project: Spark Issue

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Description: The following action fails on 3.4.1, 3.5.0, and master: {noformat} scala> val df

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Summary: Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal] (was:

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Description: The following action fails on 3.4.1, 3.5.0, and master: {noformat} scala> val df

[jira] [Commented] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785234#comment-17785234 ] Bruce Robbins commented on SPARK-45896: --- I think I have a handle on this and will

[jira] [Created] (SPARK-46189) Various Pandas functions fail in interpreted mode

2023-11-30 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46189: - Summary: Various Pandas functions fail in interpreted mode Key: SPARK-46189 URL: https://issues.apache.org/jira/browse/SPARK-46189 Project: Spark Issue Typ

[jira] [Updated] (SPARK-46189) Various Pandas functions fail in interpreted mode

2023-11-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46189: -- Description: Various Pandas functions ({{kurt}}, {{var}}, {{skew}}, {{cov}}, and {{stddev}})

[jira] [Resolved] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45644. --- Resolution: Duplicate > After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792942#comment-17792942 ] Bruce Robbins commented on SPARK-45644: --- Even though this is the original issue, I

[jira] [Created] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-06 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46289: - Summary: Exception when ordering by UDT in interpreted mode Key: SPARK-46289 URL: https://issues.apache.org/jira/browse/SPARK-46289 Project: Spark Issue Ty

[jira] [Updated] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46289: -- Affects Version/s: 3.3.3 > Exception when ordering by UDT in interpreted mode > --

[jira] [Updated] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46289: -- Priority: Minor (was: Major) > Exception when ordering by UDT in interpreted mode > -

[jira] [Commented] (SPARK-46373) Create DataFrame Bug

2023-12-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796385#comment-17796385 ] Bruce Robbins commented on SPARK-46373: --- Maybe due to this (from [the docs|https:/

[jira] [Created] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46779: - Summary: Grouping by subquery with a cached relation can fail Key: SPARK-46779 URL: https://issues.apache.org/jira/browse/SPARK-46779 Project: Spark Issue

[jira] [Updated] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46779: -- Affects Version/s: 3.5.0 3.4.2 > Grouping by subquery with a cached rel

[jira] [Updated] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46779: -- Description: Example: {noformat} create or replace temp view data(c1, c2) as values (1, 2), (1

[jira] [Commented] (SPARK-47019) AQE dynamic cache partitioning causes SortMergeJoin to result in data loss

2024-02-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816321#comment-17816321 ] Bruce Robbins commented on SPARK-47019: --- I can reproduce on my laptop using Spark

[jira] [Commented] (SPARK-47034) join between cached temp tables result in missing entries

2024-02-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17817123#comment-17817123 ] Bruce Robbins commented on SPARK-47034: --- I wonder if this is SPARK-45592 (and, rel

[jira] [Commented] (SPARK-47104) Spark SQL query fails with NullPointerException

2024-02-20 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818934#comment-17818934 ] Bruce Robbins commented on SPARK-47104: --- It's not a CSV specific issue. You can re

[jira] [Updated] (SPARK-47104) Spark SQL query fails with NullPointerException

2024-02-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47104: -- Affects Version/s: 3.5.0 3.4.2 > Spark SQL query fails with NullPointer

[jira] [Commented] (SPARK-47134) Unexpected nulls when casting decimal values in specific cases

2024-02-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819789#comment-17819789 ] Bruce Robbins commented on SPARK-47134: --- Oddly, I cannot reproduce on either 3.4.1

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821393#comment-17821393 ] Bruce Robbins commented on SPARK-47193: --- Running this in Spark 3.5.0 in local mode

[jira] [Comment Edited] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821393#comment-17821393 ] Bruce Robbins edited comment on SPARK-47193 at 2/27/24 8:48 PM: --

[jira] [Commented] (SPARK-42909) INSERT INTO with column list does not work

2023-03-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704368#comment-17704368 ] Bruce Robbins commented on SPARK-42909: --- It looks like this capability landed in 3

[jira] [Created] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

2023-03-27 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-42937: - Summary: Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled Key: SPARK-42937 URL: https://issues.apache.org/jira/browse/SPARK-42937

[jira] [Updated] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

2023-03-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-42937: -- Affects Version/s: 3.3.2 > Join with subquery in condition can fail with wholestage codegen an

[jira] [Updated] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

2023-03-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-42937: -- Affects Version/s: 3.4.0 > Join with subquery in condition can fail with wholestage codegen an

[jira] [Commented] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

2023-03-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705702#comment-17705702 ] Bruce Robbins commented on SPARK-42937: --- PR at https://github.com/apache/spark/pul

[jira] [Created] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-12 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43113: - Summary: Codegen error when full outer join's bound condition has multiple references to the same stream-side column Key: SPARK-43113 URL: https://issues.apache.org/jira/browse/

[jira] [Commented] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711614#comment-17711614 ] Bruce Robbins commented on SPARK-43113: --- PR here: https://github.com/apache/spark/

[jira] [Comment Edited] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711614#comment-17711614 ] Bruce Robbins edited comment on SPARK-43113 at 4/14/23 6:02 AM: --

[jira] [Updated] (SPARK-43149) When CTAS with USING fails to store metadata in metastore, data gets left around

2023-04-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43149: -- Summary: When CTAS with USING fails to store metadata in metastore, data gets left around (wa

[jira] [Created] (SPARK-43149) When CREATE USING fails to store metadata in metastore, data gets left around

2023-04-14 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43149: - Summary: When CREATE USING fails to store metadata in metastore, data gets left around Key: SPARK-43149 URL: https://issues.apache.org/jira/browse/SPARK-43149 Proje

[jira] [Created] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43718: - Summary: References to a specific side's key in a USING join can have wrong nullability Key: SPARK-43718 URL: https://issues.apache.org/jira/browse/SPARK-43718 Proj

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Description: Assume this data: {noformat} create or replace temp view t1 as values (1), (2), (

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Labels: correctness (was: ) > References to a specific side's key in a USING join can have wr

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Description: Assume this data: {noformat} create or replace temp view t1 as values (1), (2), (

[jira] [Commented] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725122#comment-17725122 ] Bruce Robbins commented on SPARK-43718: --- I think I have a handle on this. I will s

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Affects Version/s: 3.4.0 > References to a specific side's key in a USING join can have wrong

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Affects Version/s: 3.3.2 > References to a specific side's key in a USING join can have wrong

[jira] [Updated] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43718: -- Description: Assume this data: {noformat} create or replace temp view t1 as values (1), (2), (

[jira] [Commented] (SPARK-43718) References to a specific side's key in a USING join can have wrong nullability

2023-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725143#comment-17725143 ] Bruce Robbins commented on SPARK-43718: --- PR here: https://github.com/apache/spark/

[jira] [Created] (SPARK-43841) Non-existent column in projection of full outer join with USING results in StringIndexOutOfBoundsException

2023-05-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43841: - Summary: Non-existent column in projection of full outer join with USING results in StringIndexOutOfBoundsException Key: SPARK-43841 URL: https://issues.apache.org/jira/browse/S

[jira] [Created] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-43843: - Summary: Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError Key: SPARK-43843 URL: https://issues.apache.org/jira/browse/SPARK-43843 Project: Spark

[jira] [Updated] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-43843: -- Environment: Scala version 2.13.8 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.12) > Saving

[jira] [Commented] (SPARK-43841) Non-existent column in projection of full outer join with USING results in StringIndexOutOfBoundsException

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726980#comment-17726980 ] Bruce Robbins commented on SPARK-43841: --- PR at https://github.com/apache/spark/pul

[jira] [Commented] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726988#comment-17726988 ] Bruce Robbins commented on SPARK-43843: --- Nevermind, I had an old {{spark-avro_2.12

[jira] [Resolved] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-43843. --- Resolution: Invalid > Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError >

[jira] [Commented] (SPARK-44040) Incorrect result after count distinct

2023-06-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732163#comment-17732163 ] Bruce Robbins commented on SPARK-44040: --- It seems this can be reproduced in {{spar

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins commented on SPARK-44132: --- You may have this figured out already, bu

[jira] [Comment Edited] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins edited comment on SPARK-44132 at 6/22/23 1:51 AM: --

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735976#comment-17735976 ] Bruce Robbins commented on SPARK-44132: --- [~steven.aerts] Go for it! > nesting ful

[jira] [Created] (SPARK-44251) Potentially incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-44251: - Summary: Potentially incorrect results or NPE when full outer USING join has null key value Key: SPARK-44251 URL: https://issues.apache.org/jira/browse/SPARK-44251

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Summary: Potential for incorrect results or NPE when full outer USING join has null key value

[jira] [Commented] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738762#comment-17738762 ] Bruce Robbins commented on SPARK-44251: --- This is similar to, but not quite the sam

[jira] [Commented] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739180#comment-17739180 ] Bruce Robbins commented on SPARK-44251: --- PR can be found here: https://github.com/

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Affects Version/s: 3.4.1 > Potential for incorrect results or NPE when full outer USING join h

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Affects Version/s: 3.3.2 > Potential for incorrect results or NPE when full outer USING join h

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-07-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Labels: correctness (was: ) > Potential for incorrect results or NPE when full outer USING jo

[jira] [Created] (SPARK-36568) Missed broadcast join in V2 plan

2021-08-23 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-36568: - Summary: Missed broadcast join in V2 plan Key: SPARK-36568 URL: https://issues.apache.org/jira/browse/SPARK-36568 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-40624) A DECIMAL value with division by 0 errors in DataFrame but evaluates to NULL in SparkSQL

2022-09-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611803#comment-17611803 ] Bruce Robbins commented on SPARK-40624: --- That's not a Spark API throwing that exce

[jira] [Commented] (SPARK-40706) IllegalStateException when querying array values inside a nested struct

2022-10-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614550#comment-17614550 ] Bruce Robbins commented on SPARK-40706: --- Same as SPARK-39854? At the very least,

[jira] [Comment Edited] (SPARK-40706) IllegalStateException when querying array values inside a nested struct

2022-10-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614550#comment-17614550 ] Bruce Robbins edited comment on SPARK-40706 at 10/10/22 5:01 PM: -

[jira] [Created] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-40963: - Summary: containsNull in array type attributes is not updated from child output Key: SPARK-40963 URL: https://issues.apache.org/jira/browse/SPARK-40963 Project: Spa

[jira] [Commented] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625955#comment-17625955 ] Bruce Robbins commented on SPARK-40963: --- I'll take a stab at fixing this in the ne

[jira] [Updated] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-40963: -- Affects Version/s: 3.3.1 > containsNull in array type attributes is not updated from child out

[jira] [Updated] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-40963: -- Affects Version/s: 3.2.2 > containsNull in array type attributes is not updated from child out

[jira] [Updated] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-40963: -- Affects Version/s: 3.1.3 > containsNull in array type attributes is not updated from child out

[jira] [Updated] (SPARK-40963) containsNull in array type attributes is not updated from child output

2022-10-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-40963: -- Description: Example: {noformat} select c1, explode(c4) as c5 from ( select c1, array(c3) as

<    1   2   3   4   5   >