[jira] [Updated] (SPARK-52339) Relations may appear equal even though they are different

2025-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-52339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-52339: -- Labels: correctness (was: ) > Relations may appear equal even though they are different > ---

[jira] [Created] (SPARK-52339) Relations may appear equal even though they are different

2025-05-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-52339: - Summary: Relations may appear equal even though they are different Key: SPARK-52339 URL: https://issues.apache.org/jira/browse/SPARK-52339 Project: Spark I

[jira] [Updated] (SPARK-50091) Query fails when aggregate expression is in left-hand operand of IN-subquery

2024-11-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-50091: -- Description: Consider this query: {noformat} create or replace temp view v1(c1, c2) as values

[jira] [Updated] (SPARK-50091) Query fails when aggregate expression is in left-hand operand of IN-subquery

2024-10-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-50091: -- Affects Version/s: 3.5.2 (was: 3.5.0) > Query fails when aggregate

[jira] [Created] (SPARK-50091) Query fails when aggregate expression is in left-hand operand of IN-subquery

2024-10-23 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-50091: - Summary: Query fails when aggregate expression is in left-hand operand of IN-subquery Key: SPARK-50091 URL: https://issues.apache.org/jira/browse/SPARK-50091 Projec

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-10-02 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886449#comment-17886449 ] Bruce Robbins commented on SPARK-47193: --- PR https://github.com/apache/spark/pull/4

[jira] [Comment Edited] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-09-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879050#comment-17879050 ] Bruce Robbins edited comment on SPARK-47193 at 9/13/24 5:13 PM: --

[jira] [Commented] (SPARK-49529) Incorrect results from from_utc_timestamp function

2024-09-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879956#comment-17879956 ] Bruce Robbins commented on SPARK-49529: --- This actually matches Java 17's behavior.

[jira] [Commented] (SPARK-48950) Corrupt data from parquet scans

2024-09-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879306#comment-17879306 ] Bruce Robbins commented on SPARK-48950: --- By the way, there was a vectorization-rel

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-09-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879050#comment-17879050 ] Bruce Robbins commented on SPARK-47193: --- [~dongjoon]  I started to work on it, bu

[jira] [Commented] (SPARK-48965) toJSON produces wrong values if DecimalType information is lost in as[Product]

2024-09-02 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878658#comment-17878658 ] Bruce Robbins commented on SPARK-48965: --- [~LDVSoft] are you looking to fix this? I

[jira] [Resolved] (SPARK-45745) Extremely slow execution of sum of columns in Spark 3.4.1

2024-08-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45745. --- Resolution: Duplicate > Extremely slow execution of sum of columns in Spark 3.4.1 >

[jira] [Resolved] (SPARK-40706) IllegalStateException when querying array values inside a nested struct

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-40706. --- Resolution: Duplicate > IllegalStateException when querying array values inside a nested str

[jira] [Commented] (SPARK-45745) Extremely slow execution of sum of columns in Spark 3.4.1

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877165#comment-17877165 ] Bruce Robbins commented on SPARK-45745: --- I will close as a duplicate of SPARK-4507

[jira] [Updated] (SPARK-49261) Correlation between lit and round during grouping

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-49261: -- Target Version/s: (was: 3.5.0) > Correlation between lit and round during grouping > ---

[jira] [Updated] (SPARK-49261) Correlation between lit and round during grouping

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-49261: -- Fix Version/s: (was: 3.5.0) > Correlation between lit and round during grouping >

[jira] [Resolved] (SPARK-49350) FoldablePropagation rule and ConstantFolding rule leads to wrong aggregated result

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-49350. --- Resolution: Duplicate > FoldablePropagation rule and ConstantFolding rule leads to wrong agg

[jira] [Commented] (SPARK-49350) FoldablePropagation rule and ConstantFolding rule leads to wrong aggregated result

2024-08-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877052#comment-17877052 ] Bruce Robbins commented on SPARK-49350: --- [~Wayne Guo] Thanks for the update. Closi

[jira] [Commented] (SPARK-49350) FoldablePropagation rule and ConstantFolding rule leads to wrong aggregated result

2024-08-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876366#comment-17876366 ] Bruce Robbins commented on SPARK-49350: --- Possibly the same as SPARK-49000? > Fold

[jira] [Commented] (SPARK-49261) Correlation between lit and round during grouping

2024-08-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875657#comment-17875657 ] Bruce Robbins commented on SPARK-49261: --- {quote}It seems to be a correlation betwe

[jira] [Updated] (SPARK-48965) toJSON produces wrong values if DecimalType information is lost in as[Product]

2024-08-16 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-48965: -- Labels: correctness (was: ) > toJSON produces wrong values if DecimalType information is lost

[jira] [Comment Edited] (SPARK-48965) toJSON produces wrong values if DecimalType information is lost in as[Product]

2024-08-16 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874352#comment-17874352 ] Bruce Robbins edited comment on SPARK-48965 at 8/16/24 6:52 PM: --

[jira] [Commented] (SPARK-48965) toJSON produces wrong values if DecimalType information is lost in as[Product]

2024-08-16 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874352#comment-17874352 ] Bruce Robbins commented on SPARK-48965: --- It's not just decimals. {{toJSON}} is sim

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-06-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854211#comment-17854211 ] Bruce Robbins commented on SPARK-47193: --- I took a look at this today. This issue h

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849785#comment-17849785 ] Bruce Robbins commented on SPARK-47193: --- Thanks for the update. This issue is see

[jira] [Comment Edited] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849747#comment-17849747 ] Bruce Robbins edited comment on SPARK-48361 at 5/27/24 2:29 PM: --

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849747#comment-17849747 ] Bruce Robbins commented on SPARK-48361: --- After looking at this, I see that this is

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849095#comment-17849095 ] Bruce Robbins commented on SPARK-48361: --- I can take a look at the root cause, unle

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848787#comment-17848787 ] Bruce Robbins commented on SPARK-48361: --- Did you mean the following? {noformat} va

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848696#comment-17848696 ] Bruce Robbins commented on SPARK-48361: --- `8,9` is still present before the aggrega

[jira] [Commented] (SPARK-48361) Correctness: CSV corrupt record filter with aggregate ignored

2024-05-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848692#comment-17848692 ] Bruce Robbins commented on SPARK-48361: --- Sorry for being dense. What would the cor

[jira] [Resolved] (SPARK-47134) Unexpected nulls when casting decimal values in specific cases

2024-05-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-47134. --- Resolution: Invalid > Unexpected nulls when casting decimal values in specific cases > -

[jira] [Updated] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-03-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47633: -- Affects Version/s: 3.4.2 > Cache miss for queries using JOIN LATERAL with join condition > ---

[jira] [Updated] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-03-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47633: -- Affects Version/s: 3.5.1 > Cache miss for queries using JOIN LATERAL with join condition > ---

[jira] [Created] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-03-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-47633: - Summary: Cache miss for queries using JOIN LATERAL with join condition Key: SPARK-47633 URL: https://issues.apache.org/jira/browse/SPARK-47633 Project: Spark

[jira] [Resolved] (SPARK-47527) Cache miss for queries using With expressions

2024-03-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-47527. --- Resolution: Duplicate > Cache miss for queries using With expressions >

[jira] [Created] (SPARK-47527) Cache misses for queries using With expressions

2024-03-23 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-47527: - Summary: Cache misses for queries using With expressions Key: SPARK-47527 URL: https://issues.apache.org/jira/browse/SPARK-47527 Project: Spark Issue Type:

[jira] [Updated] (SPARK-47527) Cache miss for queries using With expressions

2024-03-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47527: -- Description: For example: {noformat} create or replace temp view v1 as select id from range(10

[jira] [Updated] (SPARK-47527) Cache miss for queries using With expressions

2024-03-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47527: -- Summary: Cache miss for queries using With expressions (was: Cache misses for queries using W

[jira] [Comment Edited] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821393#comment-17821393 ] Bruce Robbins edited comment on SPARK-47193 at 2/27/24 8:48 PM: --

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821393#comment-17821393 ] Bruce Robbins commented on SPARK-47193: --- Running this in Spark 3.5.0 in local mode

[jira] [Commented] (SPARK-47134) Unexpected nulls when casting decimal values in specific cases

2024-02-22 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819789#comment-17819789 ] Bruce Robbins commented on SPARK-47134: --- Oddly, I cannot reproduce on either 3.4.1

[jira] [Updated] (SPARK-47104) Spark SQL query fails with NullPointerException

2024-02-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-47104: -- Affects Version/s: 3.5.0 3.4.2 > Spark SQL query fails with NullPointer

[jira] [Commented] (SPARK-47104) Spark SQL query fails with NullPointerException

2024-02-20 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818934#comment-17818934 ] Bruce Robbins commented on SPARK-47104: --- It's not a CSV specific issue. You can re

[jira] [Commented] (SPARK-47034) join between cached temp tables result in missing entries

2024-02-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17817123#comment-17817123 ] Bruce Robbins commented on SPARK-47034: --- I wonder if this is SPARK-45592 (and, rel

[jira] [Commented] (SPARK-47019) AQE dynamic cache partitioning causes SortMergeJoin to result in data loss

2024-02-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816321#comment-17816321 ] Bruce Robbins commented on SPARK-47019: --- I can reproduce on my laptop using Spark

[jira] [Updated] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46779: -- Description: Example: {noformat} create or replace temp view data(c1, c2) as values (1, 2), (1

[jira] [Updated] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46779: -- Affects Version/s: 3.5.0 3.4.2 > Grouping by subquery with a cached rel

[jira] [Created] (SPARK-46779) Grouping by subquery with a cached relation can fail

2024-01-19 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46779: - Summary: Grouping by subquery with a cached relation can fail Key: SPARK-46779 URL: https://issues.apache.org/jira/browse/SPARK-46779 Project: Spark Issue

[jira] [Commented] (SPARK-46373) Create DataFrame Bug

2023-12-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796385#comment-17796385 ] Bruce Robbins commented on SPARK-46373: --- Maybe due to this (from [the docs|https:/

[jira] [Updated] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46289: -- Priority: Minor (was: Major) > Exception when ordering by UDT in interpreted mode > -

[jira] [Updated] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46289: -- Affects Version/s: 3.3.3 > Exception when ordering by UDT in interpreted mode > --

[jira] [Created] (SPARK-46289) Exception when ordering by UDT in interpreted mode

2023-12-06 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46289: - Summary: Exception when ordering by UDT in interpreted mode Key: SPARK-46289 URL: https://issues.apache.org/jira/browse/SPARK-46289 Project: Spark Issue Ty

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792942#comment-17792942 ] Bruce Robbins commented on SPARK-45644: --- Even though this is the original issue, I

[jira] [Resolved] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45644. --- Resolution: Duplicate > After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException

[jira] [Updated] (SPARK-46189) Various Pandas functions fail in interpreted mode

2023-11-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-46189: -- Description: Various Pandas functions ({{kurt}}, {{var}}, {{skew}}, {{cov}}, and {{stddev}})

[jira] [Created] (SPARK-46189) Various Pandas functions fail in interpreted mode

2023-11-30 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-46189: - Summary: Various Pandas functions fail in interpreted mode Key: SPARK-46189 URL: https://issues.apache.org/jira/browse/SPARK-46189 Project: Spark Issue Typ

[jira] [Commented] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785234#comment-17785234 ] Bruce Robbins commented on SPARK-45896: --- I think I have a handle on this and will

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Description: The following action fails on 3.4.1, 3.5.0, and master: {noformat} scala> val df

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Summary: Expression encoding fails for Seq/Map of Option[Seq/Date/Timestamp/BigDecimal] (was:

[jira] [Updated] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq]

2023-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45896: -- Description: The following action fails on 3.4.1, 3.5.0, and master: {noformat} scala> val df

[jira] [Created] (SPARK-45896) Expression encoding fails for Seq/Map of Option[Seq]

2023-11-11 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45896: - Summary: Expression encoding fails for Seq/Map of Option[Seq] Key: SPARK-45896 URL: https://issues.apache.org/jira/browse/SPARK-45896 Project: Spark Issue

[jira] [Commented] (SPARK-45797) Discrepancies in PySpark DataFrame Results When Using Window Functions and Filters

2023-11-05 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783015#comment-17783015 ] Bruce Robbins commented on SPARK-45797: --- I wonder if this is the same as SPARK-455

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781531#comment-17781531 ] Bruce Robbins commented on SPARK-45644: --- I will look into it and try to submit a f

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781494#comment-17781494 ] Bruce Robbins commented on SPARK-45644: --- OK, I can reproduce. I will take a look.

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-10-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781091#comment-17781091 ] Bruce Robbins commented on SPARK-45644: --- You can turn on display of the generated

[jira] [Updated] (SPARK-45580) Subquery changes the output schema of outer query

2023-10-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Summary: Subquery changes the output schema of outer query (was: RewritePredicateSubquery une

[jira] [Updated] (SPARK-45580) Subquery changes the output schema of the outer query

2023-10-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Summary: Subquery changes the output schema of the outer query (was: Subquery changes the out

[jira] [Resolved] (SPARK-45583) Spark SQL returning incorrect values for full outer join on keys with the same name.

2023-10-20 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45583. --- Resolution: Fixed > Spark SQL returning incorrect values for full outer join on keys with th

[jira] [Commented] (SPARK-45601) stackoverflow when executing rule ExtractWindowExpressions

2023-10-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1304#comment-1304 ] Bruce Robbins commented on SPARK-45601: --- Possibly SPARK-38666 > stackoverflow whe

[jira] [Commented] (SPARK-45583) Spark SQL returning incorrect values for full outer join on keys with the same name.

2023-10-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776783#comment-17776783 ] Bruce Robbins commented on SPARK-45583: --- Strangely, I cannot reproduce. Is some se

[jira] [Commented] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776401#comment-17776401 ] Bruce Robbins commented on SPARK-45580: --- I'll make a PR in the coming days. > Rew

[jira] [Updated] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45580: -- Description: A query can have an incorrect output schema because of a subquery. Assume this d

[jira] [Created] (SPARK-45580) RewritePredicateSubquery unexpectedly changes the output schema of certain queries

2023-10-17 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45580: - Summary: RewritePredicateSubquery unexpectedly changes the output schema of certain queries Key: SPARK-45580 URL: https://issues.apache.org/jira/browse/SPARK-45580

[jira] [Commented] (SPARK-45440) Incorrect summary counts from a CSV file

2023-10-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772724#comment-17772724 ] Bruce Robbins commented on SPARK-45440: --- I added {{inferSchema=true}} as a datasou

[jira] [Created] (SPARK-45171) GenerateExec fails to initialize non-deterministic expressions before use

2023-09-14 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45171: - Summary: GenerateExec fails to initialize non-deterministic expressions before use Key: SPARK-45171 URL: https://issues.apache.org/jira/browse/SPARK-45171 Project:

[jira] [Commented] (SPARK-44912) Spark 3.4 multi-column sum slows with many columns

2023-09-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763455#comment-17763455 ] Bruce Robbins commented on SPARK-44912: --- It looks like this was fixed with SPARK-4

[jira] [Updated] (SPARK-45106) percentile_cont gets internal error when user input fails runtime replacement's input type check

2023-09-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-45106: -- Affects Version/s: 3.3.2 > percentile_cont gets internal error when user input fails runtime

[jira] [Created] (SPARK-45106) percentile_cont gets internal error when user input fails runtime replacement's input type check

2023-09-08 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-45106: - Summary: percentile_cont gets internal error when user input fails runtime replacement's input type check Key: SPARK-45106 URL: https://issues.apache.org/jira/browse/SPARK-4510

[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44805: -- Affects Version/s: 3.4.1 > Data lost after union using > spark.sql.parquet.enableNestedColumn

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762792#comment-17762792 ] Bruce Robbins commented on SPARK-44805: --- PR here: https://github.com/apache/spark/

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-05 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762234#comment-17762234 ] Bruce Robbins commented on SPARK-44805: --- I looked at this yesterday and I think I

[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-09-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44805: -- Labels: correctness (was: ) > Data lost after union using > spark.sql.parquet.enableNestedCo

[jira] [Comment Edited] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-08-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754344#comment-17754344 ] Bruce Robbins edited comment on SPARK-44805 at 8/15/23 12:26 AM: -

[jira] [Commented] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

2023-08-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754344#comment-17754344 ] Bruce Robbins commented on SPARK-44805: --- It seems to be some weird interaction bet

[jira] [Commented] (SPARK-44477) CheckAnalysis uses error subclass as an error class

2023-07-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744314#comment-17744314 ] Bruce Robbins commented on SPARK-44477: --- PR here: https://github.com/apache/spark/

[jira] [Created] (SPARK-44477) CheckAnalysis uses error subclass as an error class

2023-07-18 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-44477: - Summary: CheckAnalysis uses error subclass as an error class Key: SPARK-44477 URL: https://issues.apache.org/jira/browse/SPARK-44477 Project: Spark Issue T

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-07-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Labels: correctness (was: ) > Potential for incorrect results or NPE when full outer USING jo

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Affects Version/s: 3.3.2 > Potential for incorrect results or NPE when full outer USING join h

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Affects Version/s: 3.4.1 > Potential for incorrect results or NPE when full outer USING join h

[jira] [Commented] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739180#comment-17739180 ] Bruce Robbins commented on SPARK-44251: --- PR can be found here: https://github.com/

[jira] [Commented] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738762#comment-17738762 ] Bruce Robbins commented on SPARK-44251: --- This is similar to, but not quite the sam

[jira] [Updated] (SPARK-44251) Potential for incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44251: -- Summary: Potential for incorrect results or NPE when full outer USING join has null key value

[jira] [Created] (SPARK-44251) Potentially incorrect results or NPE when full outer USING join has null key value

2023-06-29 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-44251: - Summary: Potentially incorrect results or NPE when full outer USING join has null key value Key: SPARK-44251 URL: https://issues.apache.org/jira/browse/SPARK-44251

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735976#comment-17735976 ] Bruce Robbins commented on SPARK-44132: --- [~steven.aerts] Go for it! > nesting ful

[jira] [Comment Edited] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins edited comment on SPARK-44132 at 6/22/23 1:51 AM: --

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

2023-06-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins commented on SPARK-44132: --- You may have this figured out already, bu

[jira] [Commented] (SPARK-44040) Incorrect result after count distinct

2023-06-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732163#comment-17732163 ] Bruce Robbins commented on SPARK-44040: --- It seems this can be reproduced in {{spar

[jira] [Resolved] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-43843. --- Resolution: Invalid > Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError >

[jira] [Commented] (SPARK-43843) Saving an AVRO file with Scala 2.13 results in NoClassDefFoundError

2023-05-28 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726988#comment-17726988 ] Bruce Robbins commented on SPARK-43843: --- Nevermind, I had an old {{spark-avro_2.12

  1   2   3   4   5   >