[jira] [Commented] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-02-03 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486654#comment-17486654 ] Prakhar Jain commented on SPARK-37980: -- Thanks [~cloud_fan] [~lian cheng] for your input. Yes -

[jira] [Comment Edited] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-26 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482845#comment-17482845 ] Prakhar Jain edited comment on SPARK-37980 at 1/27/22, 2:55 AM:

[jira] [Comment Edited] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-26 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482845#comment-17482845 ] Prakhar Jain edited comment on SPARK-37980 at 1/27/22, 2:55 AM:

[jira] [Commented] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-26 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482845#comment-17482845 ] Prakhar Jain commented on SPARK-37980: -- [~cloud_fan] I did some more investigation on this. Looks

[jira] [Comment Edited] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-25 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482142#comment-17482142 ] Prakhar Jain edited comment on SPARK-37980 at 1/26/22, 1:58 AM: Yeah -

[jira] [Commented] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-25 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482142#comment-17482142 ] Prakhar Jain commented on SPARK-37980: -- Yes - this needs implementation in the underlying

[jira] [Updated] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-22 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-37980: - Description: Spark recently added hidden metadata column support for File based datasources as

[jira] [Updated] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-22 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-37980: - Description: Spark recently added hidden metadata column support for File based datasources as

[jira] [Updated] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

2022-01-22 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-37980: - Summary: Extend METADATA column to support row indices for file based data sources (was:

[jira] [Created] (SPARK-37980) Extend METADATA column to support row indexes for file based data sources

2022-01-21 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-37980: Summary: Extend METADATA column to support row indexes for file based data sources Key: SPARK-37980 URL: https://issues.apache.org/jira/browse/SPARK-37980 Project:

[jira] [Updated] (SPARK-33758) Prune unnecessary output partitioning when the attribute is not part of output.

2020-12-14 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-33758: - Description: Consider the query: {noformat} val planned = sql( """ |

[jira] [Created] (SPARK-33758) Prune unnecessary output partitioning when the attribute is not part of output.

2020-12-11 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-33758: Summary: Prune unnecessary output partitioning when the attribute is not part of output. Key: SPARK-33758 URL: https://issues.apache.org/jira/browse/SPARK-33758

[jira] [Updated] (SPARK-33486) Collapse Partial and Final Aggregation into Complete Aggregation mode

2020-11-27 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-33486: - Issue Type: Improvement (was: Task) > Collapse Partial and Final Aggregation into Complete

[jira] [Commented] (SPARK-33486) Collapse Partial and Final Aggregation into Complete Aggregation mode

2020-11-27 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239574#comment-17239574 ] Prakhar Jain commented on SPARK-33486: -- [~dongjoon] Sure. Updating the Issue Type to Improvement.

[jira] [Created] (SPARK-33503) Refactor SortOrder class to allow multiple childrens

2020-11-20 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-33503: Summary: Refactor SortOrder class to allow multiple childrens Key: SPARK-33503 URL: https://issues.apache.org/jira/browse/SPARK-33503 Project: Spark Issue

[jira] [Created] (SPARK-33486) Collapse Partial and Final Aggregation into Complete Aggregation mode

2020-11-19 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-33486: Summary: Collapse Partial and Final Aggregation into Complete Aggregation mode Key: SPARK-33486 URL: https://issues.apache.org/jira/browse/SPARK-33486 Project: Spark

[jira] [Updated] (SPARK-33400) Normalize sameOrderExpressions in SortOrder

2020-11-17 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-33400: - Summary: Normalize sameOrderExpressions in SortOrder (was: Reduce unneeded sorts between two

[jira] [Updated] (SPARK-33399) Normalize output partitioning and sortorder with respect to aliases to avoid unneeded exchange/sort nodes

2020-11-12 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-33399: - Summary: Normalize output partitioning and sortorder with respect to aliases to avoid unneeded

[jira] [Created] (SPARK-33400) Reduce unneeded sorts between two SortMergeJoins

2020-11-09 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-33400: Summary: Reduce unneeded sorts between two SortMergeJoins Key: SPARK-33400 URL: https://issues.apache.org/jira/browse/SPARK-33400 Project: Spark Issue Type:

[jira] [Created] (SPARK-33399) Reduce unneeded exchanges after SortMerge joins

2020-11-09 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-33399: Summary: Reduce unneeded exchanges after SortMerge joins Key: SPARK-33399 URL: https://issues.apache.org/jira/browse/SPARK-33399 Project: Spark Issue Type:

[jira] [Created] (SPARK-32509) Unused DPP Filter causes issue in canonicalization and prevents reuse exchange

2020-07-31 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-32509: Summary: Unused DPP Filter causes issue in canonicalization and prevents reuse exchange Key: SPARK-32509 URL: https://issues.apache.org/jira/browse/SPARK-32509

[jira] [Created] (SPARK-32041) Exchange reuse won't work in cases when DPP, subqueries are involved

2020-06-20 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-32041: Summary: Exchange reuse won't work in cases when DPP, subqueries are involved Key: SPARK-32041 URL: https://issues.apache.org/jira/browse/SPARK-32041 Project: Spark

[jira] [Updated] (SPARK-31810) Fix recoverPartitions test in DDLSuite

2020-05-25 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-31810: - Component/s: Tests > Fix recoverPartitions test in DDLSuite >

[jira] [Created] (SPARK-31810) Fix recoverPartitions test in DDLSuite

2020-05-25 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-31810: Summary: Fix recoverPartitions test in DDLSuite Key: SPARK-31810 URL: https://issues.apache.org/jira/browse/SPARK-31810 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31618) Pushdown Distinct through Join in IntersectDistinct based on stats

2020-04-30 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-31618: Summary: Pushdown Distinct through Join in IntersectDistinct based on stats Key: SPARK-31618 URL: https://issues.apache.org/jira/browse/SPARK-31618 Project: Spark

[jira] [Updated] (SPARK-30786) Block replication is not retried on other BlockManagers when it fails on 1 of the peers

2020-02-11 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-30786: - Component/s: Spark Core > Block replication is not retried on other BlockManagers when it fails

[jira] [Commented] (SPARK-30786) Block replication is not retried on other BlockManagers when it fails on 1 of the peers

2020-02-10 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034210#comment-17034210 ] Prakhar Jain commented on SPARK-30786: -- I am working on this. > Block replication is not retried

[jira] [Created] (SPARK-30786) Block replication is not retried on other BlockManagers when it fails on 1 of the peers

2020-02-10 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-30786: Summary: Block replication is not retried on other BlockManagers when it fails on 1 of the peers Key: SPARK-30786 URL: https://issues.apache.org/jira/browse/SPARK-30786

[jira] [Updated] (SPARK-29938) Add batching in alter table add partition flow

2019-11-17 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-29938: - Description: When lot of new partitions are added by an Insert query on a partitioned

[jira] [Updated] (SPARK-29938) Add batching in alter table add partition flow

2019-11-17 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakhar Jain updated SPARK-29938: - Description: When lot of new partitions are added by an Insert query on a partitioned

[jira] [Created] (SPARK-29938) Add batching in alter table add partition flow

2019-11-17 Thread Prakhar Jain (Jira)
Prakhar Jain created SPARK-29938: Summary: Add batching in alter table add partition flow Key: SPARK-29938 URL: https://issues.apache.org/jira/browse/SPARK-29938 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-11-14 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974162#comment-16974162 ] Prakhar Jain commented on SPARK-21040: -- Hi [~holden], At Microsoft, we are also facing same issues