[jira] [Resolved] (SPARK-51720) Add Cross Join as legal in recursion of Recursive CTE

2025-04-04 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-51720. Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50308 [https://github

[jira] [Assigned] (SPARK-51720) Add Cross Join as legal in recursion of Recursive CTE

2025-04-04 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reassigned SPARK-51720: -- Assignee: Pavle Martinović > Add Cross Join as legal in recursion of Recursive CTE >

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-20 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Description: The following snipett produces inconsitent UI state: {noformat} import scala.collection

[jira] [Created] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
Peter Toth created SPARK-50356: -- Summary: Skipped stages are inconsitent on UI Key: SPARK-50356 URL: https://issues.apache.org/jira/browse/SPARK-50356 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Priority: Minor (was: Major) > Skipped stages are inconsitent on UI > -

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Description: The following snipett produces inconsitent UI state: {noformat} import scala.collection

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Description: The following snipett produces inconsitent UI state: {noformat} import scala.collection

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Attachment: (was: Screenshot 2024-11-19 at 16.07.00.png) > Skipped stages are inconsitent on UI

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Attachment: Screenshot 2024-11-19 at 16.07.20.png Screenshot 2024-11-19 at 16.07.06.p

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

2024-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Description: The following snipett produces inconsitent UI state: {noformat} import scala.collection

[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Description: {{QueryPlan.missingInput()}} calculation seems to be the root cause of {{DeduplicateRel

[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Description: {{QueryPlan.missingInput()}} calculation seems to be the root cause of {{DeduplicateRel

[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Summary: Improve missingInput calculation (was: Fix missingInput calculation) > Improve missingInp

[jira] [Created] (SPARK-47319) Fix missingInput calculation

2024-03-07 Thread Peter Toth (Jira)
Peter Toth created SPARK-47319: -- Summary: Fix missingInput calculation Key: SPARK-47319 URL: https://issues.apache.org/jira/browse/SPARK-47319 Project: Spark Issue Type: Improvement Co

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-03-01 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47217: --- Shepherd: (was: Peter Toth) > De-duplication of Relations in Joins, can result in plan resolution

[jira] [Resolved] (SPARK-45805) Eliminate magic numbers in withOrigin

2023-11-06 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-45805. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43671 [https://github

[jira] [Resolved] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-45354. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43146 [https://github

[jira] [Assigned] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reassigned SPARK-45354: -- Assignee: Peter Toth > Resolve functions bottom-up > --- > >

[jira] [Updated] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45354: --- Description: This PR proposes bottum-up resolution in  {{{}ResolveFunctions{}}}, which is much faster

[jira] [Created] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
Peter Toth created SPARK-45354: -- Summary: Resolve functions bottom-up Key: SPARK-45354 URL: https://issues.apache.org/jira/browse/SPARK-45354 Project: Spark Issue Type: Improvement Com

[jira] [Updated] (SPARK-45216) Fix non-deterministic seeded Dataset APIs

2023-09-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45216: --- Description: If we run the following example the result is the expected equal 2 columns: {noformat}

[jira] [Created] (SPARK-45216) Fix non-deterministic seeded Dataset APIs

2023-09-19 Thread Peter Toth (Jira)
Peter Toth created SPARK-45216: -- Summary: Fix non-deterministic seeded Dataset APIs Key: SPARK-45216 URL: https://issues.apache.org/jira/browse/SPARK-45216 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-11 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45112: --- Summary: Use UnresolvedFunction based resolution in SQL Dataset functions (was: Use UnresolvedFunct

[jira] [Created] (SPARK-45112) Use UnresolvedFunction in dataset functions

2023-09-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-45112: -- Summary: Use UnresolvedFunction in dataset functions Key: SPARK-45112 URL: https://issues.apache.org/jira/browse/SPARK-45112 Project: Spark Issue Type: Improveme

[jira] [Updated] (SPARK-45109) Fix eas_decrypt and ln in connect

2023-09-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45109: --- Description: The current {{eas_descrypt}} reference to {{aes_encrypt}} is clearly a bug. The {{ln}} 

[jira] [Created] (SPARK-45109) Fix eas_decrypt and ln in connect

2023-09-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-45109: -- Summary: Fix eas_decrypt and ln in connect Key: SPARK-45109 URL: https://issues.apache.org/jira/browse/SPARK-45109 Project: Spark Issue Type: Bug Compo

[jira] [Updated] (SPARK-45022) Provide context for dataset API errors

2023-08-31 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45022: --- Description: SQL failures already provide nice error context when there is a failure: {noformat} org

[jira] [Created] (SPARK-45034) Support deterministic mode function

2023-08-31 Thread Peter Toth (Jira)
Peter Toth created SPARK-45034: -- Summary: Support deterministic mode function Key: SPARK-45034 URL: https://issues.apache.org/jira/browse/SPARK-45034 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-45022) Provide context for dataset API errors

2023-08-30 Thread Peter Toth (Jira)
Peter Toth created SPARK-45022: -- Summary: Provide context for dataset API errors Key: SPARK-45022 URL: https://issues.apache.org/jira/browse/SPARK-45022 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reassigned SPARK-44934: -- Assignee: Wen Yuen Pang > PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan whe

[jira] [Updated] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44934: --- Affects Version/s: 3.3.3 > PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when call

[jira] [Resolved] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-44934. Fix Version/s: 3.3.4 3.5.0 4.0.0 3.4.2

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Fix Version/s: 3.5.0 > Fix PERCENTILE_DISC behaviour > - > >

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Affects Version/s: 3.4.0 3.3.2 3.3.1

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Affects Version/s: 3.4.1 3.3.3 (was: 3.3.0)

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Fix Version/s: 3.5.0 4.0.0 > Fix PERCENTILE_DISC behaviour >

[jira] [Commented] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756097#comment-17756097 ] Peter Toth commented on SPARK-44871: [~tgraves], sure, I've just updated it. It loo

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Description: Currently {{percentile_disc()}} returns incorrect results in some cases: E.g.: {code:j

[jira] [Created] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
Peter Toth created SPARK-44871: -- Summary: Fix PERCENTILE_DISC behaviour Key: SPARK-44871 URL: https://issues.apache.org/jira/browse/SPARK-44871 Project: Spark Issue Type: Bug Component

[jira] [Created] (SPARK-43266) Move MergeScalarSubqueries to spark-sql

2023-04-24 Thread Peter Toth (Jira)
Peter Toth created SPARK-43266: -- Summary: Move MergeScalarSubqueries to spark-sql Key: SPARK-43266 URL: https://issues.apache.org/jira/browse/SPARK-43266 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-43199) Make InlineCTE idempotent

2023-04-19 Thread Peter Toth (Jira)
Peter Toth created SPARK-43199: -- Summary: Make InlineCTE idempotent Key: SPARK-43199 URL: https://issues.apache.org/jira/browse/SPARK-43199 Project: Spark Issue Type: Bug Components: S

[jira] [Comment Edited] (SPARK-24497) ANSI SQL: Recursive query

2023-04-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17714114#comment-17714114 ] Peter Toth edited comment on SPARK-24497 at 4/19/23 2:00 PM: -

[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

2023-04-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17714114#comment-17714114 ] Peter Toth commented on SPARK-24497: I've opened a new PR: https://github.com/apache

[jira] [Created] (SPARK-43124) Dataset.show should not trigger job execution on CommandResults

2023-04-13 Thread Peter Toth (Jira)
Peter Toth created SPARK-43124: -- Summary: Dataset.show should not trigger job execution on CommandResults Key: SPARK-43124 URL: https://issues.apache.org/jira/browse/SPARK-43124 Project: Spark

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariable related changes from EquivalentExpressions

2023-03-20 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Affects Version/s: (was: 3.3.2) > Revert NamedLambdaVariable related changes from EquivalentExpr

[jira] [Resolved] (SPARK-42836) Support for recursive queries

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-42836. Resolution: Duplicate > Support for recursive queries > - > >

[jira] [Commented] (SPARK-42836) Support for recursive queries

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702138#comment-17702138 ] Peter Toth commented on SPARK-42836: Please ask on the already existing ticket to le

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariable related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Summary: Revert NamedLambdaVariable related changes from EquivalentExpressions (was: Revert NamedLa

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariables related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Summary: Revert NamedLambdaVariables related changes from EquivalentExpressions (was: Rervert Named

[jira] [Created] (SPARK-42852) Rervert NamedLambdaVariables related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
Peter Toth created SPARK-42852: -- Summary: Rervert NamedLambdaVariables related changes from EquivalentExpressions Key: SPARK-42852 URL: https://issues.apache.org/jira/browse/SPARK-42852 Project: Spark

[jira] [Updated] (SPARK-42745) Improved AliasAwareOutputExpression works with DSv2

2023-03-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42745: --- Description: After SPARK-40086 / SPARK-42049 the following, simple subselect expression containing

[jira] [Updated] (SPARK-42745) Improved AliasAwareOutputExpression works with DSv2

2023-03-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42745: --- Summary: Improved AliasAwareOutputExpression works with DSv2 (was: Fix NPE after recent AliasAwareO

[jira] [Created] (SPARK-42745) Fix NPE after recent AliasAwareOutputExpression changes

2023-03-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-42745: -- Summary: Fix NPE after recent AliasAwareOutputExpression changes Key: SPARK-42745 URL: https://issues.apache.org/jira/browse/SPARK-42745 Project: Spark Issue Typ

[jira] [Created] (SPARK-42438) Improve constraint propagation using multiTransform

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42438: -- Summary: Improve constraint propagation using multiTransform Key: SPARK-42438 URL: https://issues.apache.org/jira/browse/SPARK-42438 Project: Spark Issue Type: I

[jira] [Created] (SPARK-42436) Improve multiTransform to generate alternatives dynamically

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42436: -- Summary: Improve multiTransform to generate alternatives dynamically Key: SPARK-42436 URL: https://issues.apache.org/jira/browse/SPARK-42436 Project: Spark Issu

[jira] [Created] (SPARK-42435) Update DataTables to 1.13.2

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42435: -- Summary: Update DataTables to 1.13.2 Key: SPARK-42435 URL: https://issues.apache.org/jira/browse/SPARK-42435 Project: Spark Issue Type: Improvement Com

[jira] [Comment Edited] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685858#comment-17685858 ] Peter Toth edited comment on SPARK-42346 at 2/8/23 11:16 AM: -

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685858#comment-17685858 ] Peter Toth commented on SPARK-42346: [~ritikam], you also need to disable the "Conve

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-06 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685124#comment-17685124 ] Peter Toth commented on SPARK-42346: [~ritikam], please use the Pyspark repro in des

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-05 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684272#comment-17684272 ] Peter Toth commented on SPARK-42346: [~yumwang], [~RobinLinacre], https://github.com

[jira] [Updated] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-05 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42346: --- Affects Version/s: 3.3.0 3.4.0 3.5.0

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-04 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684184#comment-17684184 ] Peter Toth commented on SPARK-42346: Thanks for pinging me [~yumwang], this might be

[jira] [Created] (SPARK-42136) Refactor BroadcastHashJoinExec output partitioning generation

2023-01-20 Thread Peter Toth (Jira)
Peter Toth created SPARK-42136: -- Summary: Refactor BroadcastHashJoinExec output partitioning generation Key: SPARK-42136 URL: https://issues.apache.org/jira/browse/SPARK-42136 Project: Spark Is

[jira] [Created] (SPARK-42134) Fix getPartitionFiltersAndDataFilters() to handle filters without referenced attributes

2023-01-20 Thread Peter Toth (Jira)
Peter Toth created SPARK-42134: -- Summary: Fix getPartitionFiltersAndDataFilters() to handle filters without referenced attributes Key: SPARK-42134 URL: https://issues.apache.org/jira/browse/SPARK-42134 P

[jira] [Created] (SPARK-41468) Fix PlanExpression handling in EquivalentExpressions

2022-12-09 Thread Peter Toth (Jira)
Peter Toth created SPARK-41468: -- Summary: Fix PlanExpression handling in EquivalentExpressions Key: SPARK-41468 URL: https://issues.apache.org/jira/browse/SPARK-41468 Project: Spark Issue Type:

[jira] [Created] (SPARK-41367) Enable V2 file tables in read paths in session catalog

2022-12-02 Thread Peter Toth (Jira)
Peter Toth created SPARK-41367: -- Summary: Enable V2 file tables in read paths in session catalog Key: SPARK-41367 URL: https://issues.apache.org/jira/browse/SPARK-41367 Project: Spark Issue Type

[jira] [Created] (SPARK-41124) Add DSv2 PlanStabilitySuites

2022-11-13 Thread Peter Toth (Jira)
Peter Toth created SPARK-41124: -- Summary: Add DSv2 PlanStabilitySuites Key: SPARK-41124 URL: https://issues.apache.org/jira/browse/SPARK-41124 Project: Spark Issue Type: Improvement Co

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- Description: The following Pyspark script: {noformat} bin/pyspark --conf spark.io.encryption.enabled

[jira] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874 ] Peter Toth deleted comment on SPARK-40874: was (Author: petertoth): The following Pyspark script: {noformat} bin/pyspark --conf spark.io.encryption.enabled=true ... bar = {"a": "aa", "b": "

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- The following Pyspark script: {noformat} bin/pyspark --conf spark.io.encryption.enabled=true ... bar =

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- Summary: Fix broadcasts in Python UDFs when encryption is enabled (was: Fix Python UDFs with broadc

[jira] [Created] (SPARK-40874) Fix Python UDFs with broadcasts when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
Peter Toth created SPARK-40874: -- Summary: Fix Python UDFs with broadcasts when encryption is enabled Key: SPARK-40874 URL: https://issues.apache.org/jira/browse/SPARK-40874 Project: Spark Issue

[jira] [Updated] (SPARK-40599) Add multiTransform methods to TreeNode to generate alternatives

2022-09-28 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40599: --- Summary: Add multiTransform methods to TreeNode to generate alternatives (was: Add multiTransform m

[jira] [Created] (SPARK-40599) Add multiTransform methods to TreeNode to generate alternative transformations

2022-09-28 Thread Peter Toth (Jira)
Peter Toth created SPARK-40599: -- Summary: Add multiTransform methods to TreeNode to generate alternative transformations Key: SPARK-40599 URL: https://issues.apache.org/jira/browse/SPARK-40599 Project: S

[jira] [Updated] (SPARK-40259) Support Parquet DSv2 in subquery plan merge

2022-08-29 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40259: --- Description: We could improve SPARK-34079 with DSv2 support. (was: We could improve SPARK-34079 to

[jira] [Created] (SPARK-40259) Support Parquet DSv2 in subquery plan merge

2022-08-29 Thread Peter Toth (Jira)
Peter Toth created SPARK-40259: -- Summary: Support Parquet DSv2 in subquery plan merge Key: SPARK-40259 URL: https://issues.apache.org/jira/browse/SPARK-40259 Project: Spark Issue Type: Improveme

[jira] [Created] (SPARK-40247) Fix BitSet equality check

2022-08-28 Thread Peter Toth (Jira)
Peter Toth created SPARK-40247: -- Summary: Fix BitSet equality check Key: SPARK-40247 URL: https://issues.apache.org/jira/browse/SPARK-40247 Project: Spark Issue Type: Bug Components: S

[jira] [Updated] (SPARK-40245) Fix FileScan equality check when partition or data filter columns are not read

2022-08-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40245: --- Summary: Fix FileScan equality check when partition or data filter columns are not read (was: Fix F

[jira] [Created] (SPARK-40245) Fix FileScan canonicalization when partition or data filter columns are not read

2022-08-27 Thread Peter Toth (Jira)
Peter Toth created SPARK-40245: -- Summary: Fix FileScan canonicalization when partition or data filter columns are not read Key: SPARK-40245 URL: https://issues.apache.org/jira/browse/SPARK-40245 Project:

[jira] [Updated] (SPARK-40193) Merge subquery plans with different filters

2022-08-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40193: --- Summary: Merge subquery plans with different filters (was: Merge different filters when merging sub

[jira] [Created] (SPARK-40193) Merge different filters when merging subquery plans

2022-08-23 Thread Peter Toth (Jira)
Peter Toth created SPARK-40193: -- Summary: Merge different filters when merging subquery plans Key: SPARK-40193 URL: https://issues.apache.org/jira/browse/SPARK-40193 Project: Spark Issue Type: I

[jira] [Created] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Peter Toth (Jira)
Peter Toth created SPARK-40086: -- Summary: Improve AliasAwareOutputPartitioning to take all aliases into account Key: SPARK-40086 URL: https://issues.apache.org/jira/browse/SPARK-40086 Project: Spark

[jira] [Created] (SPARK-38717) Handle Hive's bucket spec case preserving behaviour

2022-03-31 Thread Peter Toth (Jira)
Peter Toth created SPARK-38717: -- Summary: Handle Hive's bucket spec case preserving behaviour Key: SPARK-38717 URL: https://issues.apache.org/jira/browse/SPARK-38717 Project: Spark Issue Type: B

[jira] [Commented] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2022-03-25 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512270#comment-17512270 ] Peter Toth commented on SPARK-26639: [~stubartmess], that's a different issue but it

[jira] [Resolved] (SPARK-28299) Evaluation of multiple CTE uses

2021-11-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-28299. Resolution: Duplicate > Evaluation of multiple CTE uses > - > >

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

2021-11-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447980#comment-17447980 ] Peter Toth commented on SPARK-37259: I've opened a PR: [https://github.com/apache/sp

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

2021-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446554#comment-17446554 ] Peter Toth commented on SPARK-37259: [~KevinAppelBofa], how about adding a new `with

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419388#comment-17419388 ] Peter Toth commented on SPARK-35672: I put up a revert PR: https://github.com/apache

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419285#comment-17419285 ] Peter Toth commented on SPARK-35672: [~xkrogen], [~tgraves], unfortunately, I think

[jira] [Commented] (SPARK-36065) date_trunc returns incorrect output

2021-07-30 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390421#comment-17390421 ] Peter Toth commented on SPARK-36065: I think the output is correct as there was a ti

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-12 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Issue Type: Improvement (was: Bug) > EquivalentExpressions fixes and improvements > ---

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Description: Currently `EquivalentExpressions` has 2 issues: - identifying common expressions in con

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Issue Type: Bug (was: Improvement) > EquivalentExpressions fixes and improvements > ---

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Priority: Major (was: Minor) > EquivalentExpressions fixes and improvements > -

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Description: Fixes an issue with identifying common expressions in conditional expressions (a side

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Summary: EquivalentExpressions fixes and improvements (was: SubExpr elimination should include comm

[jira] [Created] (SPARK-36073) SubExpr elimination should include common child exprs of conditional expressions

2021-07-09 Thread Peter Toth (Jira)
Peter Toth created SPARK-36073: -- Summary: SubExpr elimination should include common child exprs of conditional expressions Key: SPARK-36073 URL: https://issues.apache.org/jira/browse/SPARK-36073 Project:

[jira] [Created] (SPARK-35855) Unify reuse map data structures in non-AQE and AQE rules

2021-06-22 Thread Peter Toth (Jira)
Peter Toth created SPARK-35855: -- Summary: Unify reuse map data structures in non-AQE and AQE rules Key: SPARK-35855 URL: https://issues.apache.org/jira/browse/SPARK-35855 Project: Spark Issue Ty

[jira] [Created] (SPARK-35798) Fix SparkPlan.sqlContext usage

2021-06-17 Thread Peter Toth (Jira)
Peter Toth created SPARK-35798: -- Summary: Fix SparkPlan.sqlContext usage Key: SPARK-35798 URL: https://issues.apache.org/jira/browse/SPARK-35798 Project: Spark Issue Type: Bug Componen

[jira] [Commented] (SPARK-34801) java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition

2021-03-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306093#comment-17306093 ] Peter Toth commented on SPARK-34801: Yes it is. Please use CDS 3 (Cloudera Distribut

  1   2   3   >