[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-27 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/43854] in

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/43854] in

[jira] [Created] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-26 Thread Asif (Jira)
Asif created SPARK-47609: Summary: CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan Key: SPARK-47609 URL: https://issues.apache.org/jira/browse/SPARK-47609 Project: Spark

[jira] [Comment Edited] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831116#comment-17831116 ] Asif edited comment on SPARK-26708 at 3/27/24 12:58 AM: I believe the current

[jira] [Comment Edited] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831117#comment-17831117 ] Asif edited comment on SPARK-26708 at 3/27/24 12:54 AM: Towards that please take

[jira] [Commented] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831116#comment-17831116 ] Asif commented on SPARK-26708: -- I believe the current caching logic is suboptimal and accordingly the bug

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Labels: pull-request-available (was: ) > Datasets involving self joins behave in an inconsistent and

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when

[jira] [Commented] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824877#comment-17824877 ] Asif commented on SPARK-47320: -- Opened following PR

[jira] [Commented] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-07 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824589#comment-17824589 ] Asif commented on SPARK-47320: -- will be linking the bug to an open PR > Datasets involving self joins

[jira] [Created] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-07 Thread Asif (Jira)
Asif created SPARK-47320: Summary: Datasets involving self joins behave in an inconsistent and unintuitive manner Key: SPARK-47320 URL: https://issues.apache.org/jira/browse/SPARK-47320 Project: Spark

[jira] [Commented] (SPARK-39441) Speed up DeduplicateRelations

2024-03-06 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824102#comment-17824102 ] Asif commented on SPARK-39441: -- this issue should be resolved by the PR for ticket

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

2024-03-06 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824102#comment-17824102 ] Asif edited comment on SPARK-39441 at 3/6/24 5:33 PM: -- this issue should be

[jira] [Comment Edited] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823510#comment-17823510 ] Asif edited comment on SPARK-33152 at 3/5/24 6:43 PM: -- [~tedjenks] .. Unfortunately

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823512#comment-17823512 ] Asif commented on SPARK-33152: -- other than using my PR, the safe option would be to disable constraint

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823510#comment-17823510 ] Asif commented on SPARK-33152: -- [~tedjenks] .. Unfortunately I am not a committer. As part of workday , I

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823344#comment-17823344 ] Asif commented on SPARK-33152: -- [~tedjenks]  The issue has always been there  because of the way constraint

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of  nested joins involving repetition of relation, the projected columns

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of self join queries or nested joins involving repetition of relation,

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of nested self join queries, the projected columns when passed to the

[jira] [Created] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
Asif created SPARK-47217: Summary: De-duplication of Relations in Joins, can result in plan resolution failure Key: SPARK-47217 URL: https://issues.apache.org/jira/browse/SPARK-47217 Project: Spark

[jira] [Updated] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-46671: - Description: while bring my old PR which uses a different approach to the ConstraintPropagation algorithm (

[jira] [Reopened] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif reopened SPARK-46671: -- After further analysis , I believe , that what I said originally in the ticket is valid and that the code Does

[jira] [Resolved] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-46671. -- Resolution: Not A Bug > InferFiltersFromConstraint rule is creating a redundant filter >

[jira] [Commented] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805434#comment-17805434 ] Asif commented on SPARK-46671: -- on further thoughts , I am wrong.. There should be 2 separate isNotNull

[jira] [Commented] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805435#comment-17805435 ] Asif commented on SPARK-46671: -- so closing the ticket > InferFiltersFromConstraint rule is creating a

[jira] [Created] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-10 Thread Asif (Jira)
Asif created SPARK-46671: Summary: InferFiltersFromConstraint rule is creating a redundant filter Key: SPARK-46671 URL: https://issues.apache.org/jira/browse/SPARK-46671 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-01-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Description: Though documentation clearly recommends to add all columns in a single shot, but in reality is

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-01-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Summary: SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation (was: Abusing

[jira] [Updated] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Priority: Minor (was: Major) > Abusing DataSet.withColumn can cause huge tree with severe perf degradation >

[jira] [Commented] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786941#comment-17786941 ] Asif commented on SPARK-45959: -- will create a PR for the same.. > Abusing DataSet.withColumn can cause

[jira] [Created] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
Asif created SPARK-45959: Summary: Abusing DataSet.withColumn can cause huge tree with severe perf degradation Key: SPARK-45959 URL: https://issues.apache.org/jira/browse/SPARK-45959 Project: Spark

[jira] [Commented] (SPARK-45943) DataSourceV2Relation.computeStats throws IllegalStateException in test mode

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786652#comment-17786652 ] Asif commented on SPARK-45943: -- thanks [~wforget] for the input.. if you have solution pls open PR, else I

[jira] [Updated] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45866: - Labels: pull-request-available (was: ) > Reuse of exchange in AQE does not happen when run time filters are

[jira] [Created] (SPARK-45943) DataSourceV2Relation.computeStats throws IllegalStateException in test mode

2023-11-15 Thread Asif (Jira)
Asif created SPARK-45943: Summary: DataSourceV2Relation.computeStats throws IllegalStateException in test mode Key: SPARK-45943 URL: https://issues.apache.org/jira/browse/SPARK-45943 Project: Spark

[jira] [Closed] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif closed SPARK-45924. this is not a bug > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not > equivalent with

[jira] [Closed] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif closed SPARK-45925. this is not an issue > SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec > causing re-use

[jira] [Resolved] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-45924. -- Resolution: Not A Bug > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not > equivalent

[jira] [Resolved] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-45925. -- Resolution: Not A Problem > SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec >

[jira] [Commented] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786155#comment-17786155 ] Asif commented on SPARK-45866: -- Now that the other PRs on which this ticket itself is dependent are

[jira] [Updated] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45925: - Labels: pull-request-available (was: ) > SubqueryBroadcastExec is not equivalent with

[jira] [Updated] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45924: - Labels: pull-request-available (was: ) > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is

[jira] [Created] (SPARK-45926) The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45926: Summary: The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly Key: SPARK-45926 URL: https://issues.apache.org/jira/browse/SPARK-45926

[jira] [Created] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45925: Summary: SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE Key: SPARK-45925 URL:

[jira] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658 ] Asif deleted comment on SPARK-45658: -- was (Author: ashahid7): I also think that during canonicalization of DynamicPruningSubquery, the pruning key's canonicalization should be done on the basis of

[jira] [Updated] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45924: - Description: while writing bug test for

[jira] [Created] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45924: Summary: Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec Key: SPARK-45924 URL: https://issues.apache.org/jira/browse/SPARK-45924

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: (was: Peter Toth) > Minimizing calls to HiveMetaStore layer for getting partitions, when tables >

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Affects Version/s: 3.5.0 (was: 2.4.0) (was: 3.0.1)

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 3.5.0 (was: 4.0.0) > Minimizing calls to HiveMetaStore layer for

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 3.5.0 (was: 3.5.1) > SPIP: Improving performance of

[jira] [Created] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-09 Thread Asif (Jira)
Asif created SPARK-45866: Summary: Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg ) Key: SPARK-45866 URL:

[jira] [Commented] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784567#comment-17784567 ] Asif commented on SPARK-45658: -- I also think that during canonicalization of DynamicPruningSubquery, the

[jira] [Commented] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784282#comment-17784282 ] Asif commented on SPARK-44662: -- The changes for iceberg which support broadcast-var-pushdown are present in

[jira] [Commented] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782927#comment-17782927 ] Asif commented on SPARK-44662: -- The majority of file changes are due to additional tpcds tests for iceberg.

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Attachment: perf results broadcast var pushdown - Partitioned TPCDS.pdf > SPIP: Improving performance of

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 3.5.1 (was: 3.3.3) > SPIP: Improving performance of

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Commented] (SPARK-36786) SPIP: Improving the compile time performance, by improving a couple of rules, from 24 hrs to under 8 minutes

2023-11-01 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781976#comment-17781976 ] Asif commented on SPARK-36786: -- I had put this on back burner as my changes were on 3.2, so I have to do a

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Description: The canonicalization of (buildKeys: Seq[Expression]) in the class DynamicPruningSubquery is

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Priority: Major (was: Critical) > Canonicalization of DynamicPruningSubquery is broken >

[jira] [Created] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
Asif created SPARK-45658: Summary: Canonicalization of DynamicPruningSubquery is broken Key: SPARK-45658 URL: https://issues.apache.org/jira/browse/SPARK-45658 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-10-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 4.0.0 (was: 3.5.1) > Minimizing calls to HiveMetaStore layer for

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-29 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Description: In the rule PruneFileSourcePartitions where the CatalogFileIndex gets converted to

[jira] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-29 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373 ] Asif deleted comment on SPARK-45373: -- was (Author: ashahid7): Will be generating a PR for this. > Minimizing calls to HiveMetaStore layer for getting partitions, when tables > are repeated >

[jira] [Commented] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770220#comment-17770220 ] Asif commented on SPARK-45373: -- Will be generating a PR for this. > Minimizing calls to HiveMetaStore

[jira] [Created] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-28 Thread Asif (Jira)
Asif created SPARK-45373: Summary: Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated Key: SPARK-45373 URL: https://issues.apache.org/jira/browse/SPARK-45373 Project:

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On

[jira] [Created] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-08-03 Thread Asif (Jira)
Asif created SPARK-44662: Summary: SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns Key: SPARK-44662 URL: https://issues.apache.org/jira/browse/SPARK-44662

[jira] [Resolved] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-43112. -- Resolution: Not A Bug > Spark may use a column other than the actual specified partitioning column > for

[jira] [Commented] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711607#comment-17711607 ] Asif commented on SPARK-43112: -- Open a WIP PR [SPARK-43112|https://github.com/apache/spark/pull/40765/]

[jira] [Updated] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-43112: - Description: The class org.apache.spark.sql.catalyst.catalog.HiveTableRelation has its output method

[jira] [Created] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-12 Thread Asif (Jira)
Asif created SPARK-43112: Summary: Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables Key: SPARK-43112 URL:

[jira] [Commented] (SPARK-41141) avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

2022-11-18 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635989#comment-17635989 ] Asif commented on SPARK-41141: -- Opened the following PR

[jira] [Updated] (SPARK-41141) avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

2022-11-18 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-41141: - Priority: Minor (was: Major) > avoid introducing a new aggregate expression in the analysis phase when >

[jira] [Updated] (SPARK-41141) avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

2022-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-41141: - Description: Currently the  analyzer phase rules on subquery referencing the aggregate expression in outer

[jira] [Created] (SPARK-41141) avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

2022-11-14 Thread Asif (Jira)
Asif created SPARK-41141: Summary: avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it Key: SPARK-41141 URL: https://issues.apache.org/jira/browse/SPARK-41141

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2022-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606817#comment-17606817 ] Asif commented on SPARK-33152: -- Added a test *CompareNewAndOldConstraintsSuite* in the PR which when run on

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2022-09-13 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Shepherd: Wenchen Fan (was: Arnaud Doucet) Description: h2. Q1. What are you trying to do? Articulate

[jira] [Updated] (SPARK-40362) Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

2022-09-07 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-40362: - Description: In the canonicalization code which is now in two stages, canonicalization involving Commutative

  1   2   >