[jira] [Commented] (SPARK-44167) SPIP: Stored Procedures API for Catalogs

2024-04-19 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839118#comment-17839118 ] Anton Okolnychyi commented on SPARK-44167: -- I've added a design doc to the description and will

[jira] [Updated] (SPARK-44167) SPIP: Stored Procedures API for Catalogs

2024-04-19 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-44167: - Shepherd: L. C. Hsieh Description: Similar to engines like Trino, it would be

[jira] [Updated] (SPARK-44405) Reduce code duplication in group-based DELETE and MERGE tests

2023-07-12 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-44405: - Summary: Reduce code duplication in group-based DELETE and MERGE tests (was: Reduce

[jira] [Created] (SPARK-44405) Reduce code duplication in row-level tests

2023-07-12 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44405: Summary: Reduce code duplication in row-level tests Key: SPARK-44405 URL: https://issues.apache.org/jira/browse/SPARK-44405 Project: Spark Issue

[jira] [Created] (SPARK-44403) Add docs for MergeRows node

2023-07-12 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44403: Summary: Add docs for MergeRows node Key: SPARK-44403 URL: https://issues.apache.org/jira/browse/SPARK-44403 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44402) Add tests for schema pruning in delta-based DELETEs

2023-07-12 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44402: Summary: Add tests for schema pruning in delta-based DELETEs Key: SPARK-44402 URL: https://issues.apache.org/jira/browse/SPARK-44402 Project: Spark

[jira] [Created] (SPARK-44392) Add tests for schema pruning in delta-based UPDATEs

2023-07-12 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44392: Summary: Add tests for schema pruning in delta-based UPDATEs Key: SPARK-44392 URL: https://issues.apache.org/jira/browse/SPARK-44392 Project: Spark

[jira] [Created] (SPARK-44360) Support schema pruning in delta-based MERGE operations

2023-07-10 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44360: Summary: Support schema pruning in delta-based MERGE operations Key: SPARK-44360 URL: https://issues.apache.org/jira/browse/SPARK-44360 Project: Spark

[jira] [Commented] (SPARK-43203) Fix DROP table behavior in session catalog

2023-06-27 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737894#comment-17737894 ] Anton Okolnychyi commented on SPARK-43203: -- I unfortunately created this initially as

[jira] [Updated] (SPARK-43203) Fix DROP table behavior in session catalog

2023-06-27 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43203: - Issue Type: Bug (was: Improvement) > Fix DROP table behavior in session catalog >

[jira] [Created] (SPARK-44167) Stored procedure API

2023-06-24 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44167: Summary: Stored procedure API Key: SPARK-44167 URL: https://issues.apache.org/jira/browse/SPARK-44167 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44148) Validate NO_BROADCAST_AND_REPLICATION hint is never set on both sides

2023-06-22 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44148: Summary: Validate NO_BROADCAST_AND_REPLICATION hint is never set on both sides Key: SPARK-44148 URL: https://issues.apache.org/jira/browse/SPARK-44148

[jira] [Created] (SPARK-44139) Discard completely pushed down filters in group-based MERGE operations

2023-06-21 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44139: Summary: Discard completely pushed down filters in group-based MERGE operations Key: SPARK-44139 URL: https://issues.apache.org/jira/browse/SPARK-44139

[jira] [Updated] (SPARK-44138) Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions

2023-06-21 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-44138: - Summary: Prohibit non-deterministic expressions, subqueries and aggregates in MERGE

[jira] [Created] (SPARK-44138) Prohibit non-deterministic conditions, subqueries and aggregates in MERGE conditions

2023-06-21 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44138: Summary: Prohibit non-deterministic conditions, subqueries and aggregates in MERGE conditions Key: SPARK-44138 URL: https://issues.apache.org/jira/browse/SPARK-44138

[jira] [Created] (SPARK-44043) Reuse main scan exchange in group-based UPDATEs

2023-06-13 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44043: Summary: Reuse main scan exchange in group-based UPDATEs Key: SPARK-44043 URL: https://issues.apache.org/jira/browse/SPARK-44043 Project: Spark

[jira] [Created] (SPARK-44013) Add benchmark for MergeRowsExec

2023-06-08 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44013: Summary: Add benchmark for MergeRowsExec Key: SPARK-44013 URL: https://issues.apache.org/jira/browse/SPARK-44013 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44000) Add join hint to disable broadcasting and replicating one side of a join

2023-06-07 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-44000: Summary: Add join hint to disable broadcasting and replicating one side of a join Key: SPARK-44000 URL: https://issues.apache.org/jira/browse/SPARK-44000

[jira] [Created] (SPARK-43975) DataSource V2: Handle UPDATE commands for group-based sources

2023-06-05 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43975: Summary: DataSource V2: Handle UPDATE commands for group-based sources Key: SPARK-43975 URL: https://issues.apache.org/jira/browse/SPARK-43975 Project: Spark

[jira] [Updated] (SPARK-43975) DataSource V2: Handle UPDATE commands for group-based sources

2023-06-05 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43975: - Description: We need to handle UPDATE commands for group-based sources. (was: We need

[jira] [Created] (SPARK-43963) DataSource V2: Handle MERGE commands for group-based sources

2023-06-04 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43963: Summary: DataSource V2: Handle MERGE commands for group-based sources Key: SPARK-43963 URL: https://issues.apache.org/jira/browse/SPARK-43963 Project: Spark

[jira] [Updated] (SPARK-43959) Make RowLevelOperationSuiteBase and AlignAssignmentsSuite abstract

2023-06-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43959: - Description: Make RowLevelOperationSuiteBase and AlignAssignmentsSuite abstract. (was:

[jira] [Updated] (SPARK-43959) Make RowLevelOperationSuiteBase and AlignAssignmentsSuite abstract

2023-06-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43959: - Summary: Make RowLevelOperationSuiteBase and AlignAssignmentsSuite abstract (was: Make

[jira] [Created] (SPARK-43959) Make RowLevelOperationSuiteBase abstract

2023-06-03 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43959: Summary: Make RowLevelOperationSuiteBase abstract Key: SPARK-43959 URL: https://issues.apache.org/jira/browse/SPARK-43959 Project: Spark Issue Type:

[jira] [Created] (SPARK-43885) DataSource V2: Handle MERGE commands for delta-based sources

2023-05-30 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43885: Summary: DataSource V2: Handle MERGE commands for delta-based sources Key: SPARK-43885 URL: https://issues.apache.org/jira/browse/SPARK-43885 Project: Spark

[jira] [Created] (SPARK-43775) DataSource V2: Allow representing updates as deletes and inserts

2023-05-24 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43775: Summary: DataSource V2: Allow representing updates as deletes and inserts Key: SPARK-43775 URL: https://issues.apache.org/jira/browse/SPARK-43775 Project:

[jira] [Updated] (SPARK-43324) DataSource V2: Handle UPDATE commands for delta-based sources

2023-05-01 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43324: - Parent: SPARK-35801 Issue Type: Sub-task (was: Improvement) > DataSource V2:

[jira] [Updated] (SPARK-43324) DataSource V2: Handle UPDATE commands for delta-based sources

2023-05-01 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43324: - Issue Type: Improvement (was: Bug) > DataSource V2: Handle UPDATE commands for

[jira] [Created] (SPARK-43324) DataSource V2: Handle UPDATE commands for delta-based sources

2023-04-28 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43324: Summary: DataSource V2: Handle UPDATE commands for delta-based sources Key: SPARK-43324 URL: https://issues.apache.org/jira/browse/SPARK-43324 Project: Spark

[jira] [Created] (SPARK-43268) Use proper error classes when exceptions are constructed with a message

2023-04-24 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43268: Summary: Use proper error classes when exceptions are constructed with a message Key: SPARK-43268 URL: https://issues.apache.org/jira/browse/SPARK-43268

[jira] [Created] (SPARK-43204) Align MERGE assignments with table attributes

2023-04-19 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43204: Summary: Align MERGE assignments with table attributes Key: SPARK-43204 URL: https://issues.apache.org/jira/browse/SPARK-43204 Project: Spark Issue

[jira] [Created] (SPARK-43203) Fix DROP table behavior in session catalog

2023-04-19 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43203: Summary: Fix DROP table behavior in session catalog Key: SPARK-43203 URL: https://issues.apache.org/jira/browse/SPARK-43203 Project: Spark Issue

[jira] [Updated] (SPARK-43203) Fix DROP table behavior in session catalog

2023-04-19 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-43203: - Description: DROP table behavior is not working correctly in 3.4.0 because we always

[jira] [Created] (SPARK-43088) Respect RequiresDistributionAndOrdering in CTAS/RTAS

2023-04-10 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43088: Summary: Respect RequiresDistributionAndOrdering in CTAS/RTAS Key: SPARK-43088 URL: https://issues.apache.org/jira/browse/SPARK-43088 Project: Spark

[jira] [Commented] (SPARK-43033) Avoid task retries due to AssertNotNull checks

2023-04-06 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709455#comment-17709455 ] Anton Okolnychyi commented on SPARK-43033: -- I haven't started working on it, so feel free to

[jira] [Commented] (SPARK-43041) Restore constructors of exceptions for compatibility in connector API

2023-04-05 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709097#comment-17709097 ] Anton Okolnychyi commented on SPARK-43041: -- https://github.com/apache/spark/pull/40679 >

[jira] [Created] (SPARK-43034) Improve error message in AssertNotNull by including combined column path

2023-04-04 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43034: Summary: Improve error message in AssertNotNull by including combined column path Key: SPARK-43034 URL: https://issues.apache.org/jira/browse/SPARK-43034

[jira] [Created] (SPARK-43033) Avoid task retries due to AssertNotNull checks

2023-04-04 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-43033: Summary: Avoid task retries due to AssertNotNull checks Key: SPARK-43033 URL: https://issues.apache.org/jira/browse/SPARK-43033 Project: Spark Issue

[jira] [Created] (SPARK-42997) TableOutputResolver must use correct column paths in error messages for arrays and maps

2023-03-31 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42997: Summary: TableOutputResolver must use correct column paths in error messages for arrays and maps Key: SPARK-42997 URL: https://issues.apache.org/jira/browse/SPARK-42997

[jira] [Created] (SPARK-42855) Migrate nullability write checks to runtime for V2 tables

2023-03-19 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42855: Summary: Migrate nullability write checks to runtime for V2 tables Key: SPARK-42855 URL: https://issues.apache.org/jira/browse/SPARK-42855 Project: Spark

[jira] [Created] (SPARK-42779) Allow V2 writes to indicate advisory partition size

2023-03-14 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42779: Summary: Allow V2 writes to indicate advisory partition size Key: SPARK-42779 URL: https://issues.apache.org/jira/browse/SPARK-42779 Project: Spark

[jira] [Updated] (SPARK-42779) Allow V2 writes to indicate advisory partition size

2023-03-14 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-42779: - Description: Data sources may request a particular distribution and ordering of data

[jira] [Updated] (SPARK-42611) Insert char/varchar length checks for inner fields during resolution

2023-02-27 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-42611: - Affects Version/s: 3.3.2 3.3.1 3.3.0

[jira] [Created] (SPARK-42611) Insert char/varchar length checks for inner fields during resolution

2023-02-27 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42611: Summary: Insert char/varchar length checks for inner fields during resolution Key: SPARK-42611 URL: https://issues.apache.org/jira/browse/SPARK-42611

[jira] [Created] (SPARK-42608) Use full column names for inner fields in resolution errors

2023-02-27 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42608: Summary: Use full column names for inner fields in resolution errors Key: SPARK-42608 URL: https://issues.apache.org/jira/browse/SPARK-42608 Project: Spark

[jira] [Commented] (SPARK-35801) SPIP: Row-level operations in Data Source V2

2023-01-21 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679464#comment-17679464 ] Anton Okolnychyi commented on SPARK-35801: -- We should probably keep it open even beyond 3.4 as

[jira] [Created] (SPARK-42151) Align UPDATE assignments with table attributes

2023-01-21 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-42151: Summary: Align UPDATE assignments with table attributes Key: SPARK-42151 URL: https://issues.apache.org/jira/browse/SPARK-42151 Project: Spark Issue

[jira] [Created] (SPARK-40551) DataSource V2: Add APIs for delta-based row-level operations

2022-09-25 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-40551: Summary: DataSource V2: Add APIs for delta-based row-level operations Key: SPARK-40551 URL: https://issues.apache.org/jira/browse/SPARK-40551 Project: Spark

[jira] [Created] (SPARK-40550) DataSource V2: Handle DELETE commands for delta-based sources

2022-09-25 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-40550: Summary: DataSource V2: Handle DELETE commands for delta-based sources Key: SPARK-40550 URL: https://issues.apache.org/jira/browse/SPARK-40550 Project: Spark

[jira] [Created] (SPARK-40295) Allow v2 functions with literal args in write distribution and ordering

2022-08-31 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-40295: Summary: Allow v2 functions with literal args in write distribution and ordering Key: SPARK-40295 URL: https://issues.apache.org/jira/browse/SPARK-40295

[jira] [Created] (SPARK-38977) Fix schema pruning with correlated subqueries

2022-04-20 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-38977: Summary: Fix schema pruning with correlated subqueries Key: SPARK-38977 URL: https://issues.apache.org/jira/browse/SPARK-38977 Project: Spark Issue

[jira] [Created] (SPARK-38959) DataSource V2: Support runtime group filtering in row-level commands

2022-04-19 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-38959: Summary: DataSource V2: Support runtime group filtering in row-level commands Key: SPARK-38959 URL: https://issues.apache.org/jira/browse/SPARK-38959

[jira] [Created] (SPARK-38626) Make condition in DeleteFromTable required

2022-03-22 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-38626: Summary: Make condition in DeleteFromTable required Key: SPARK-38626 URL: https://issues.apache.org/jira/browse/SPARK-38626 Project: Spark Issue

[jira] [Updated] (SPARK-38085) DataSource V2: Handle DELETE commands for group-based sources

2022-03-22 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-38085: - Parent: SPARK-35801 Issue Type: Sub-task (was: Improvement) > DataSource V2:

[jira] [Created] (SPARK-38625) DataSource V2: Add APIs for group-based row-level operations

2022-03-22 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-38625: Summary: DataSource V2: Add APIs for group-based row-level operations Key: SPARK-38625 URL: https://issues.apache.org/jira/browse/SPARK-38625 Project: Spark

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2022-02-07 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17488520#comment-17488520 ] Anton Okolnychyi commented on SPARK-36665: -- Thank you [~kazuyukitanimura] [~viirya]! > Add

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2022-02-04 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17487235#comment-17487235 ] Anton Okolnychyi commented on SPARK-36665: -- Thanks for the prompt reply, [~kazuyukitanimura]!

[jira] [Comment Edited] (SPARK-36665) Add more Not operator optimizations

2022-02-04 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17487200#comment-17487200 ] Anton Okolnychyi edited comment on SPARK-36665 at 2/4/22, 5:47 PM: ---

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2022-02-04 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17487200#comment-17487200 ] Anton Okolnychyi commented on SPARK-36665: -- [~kazuyukitanimura] {{RewritePredicateSubquery}}

[jira] [Comment Edited] (SPARK-36665) Add more Not operator optimizations

2022-02-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486819#comment-17486819 ] Anton Okolnychyi edited comment on SPARK-36665 at 2/4/22, 5:16 AM: ---

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2022-02-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486819#comment-17486819 ] Anton Okolnychyi commented on SPARK-36665: -- The proposed logic seems to behave correctly for IN

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2022-02-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486817#comment-17486817 ] Anton Okolnychyi commented on SPARK-36665: -- [~kazuyukitanimura] [~viirya] I am afraid this rule

[jira] [Updated] (SPARK-36665) Add more Not operator optimizations

2022-02-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-36665: - Attachment: Pasted Graphic 3.png > Add more Not operator optimizations >

[jira] [Created] (SPARK-38085) DataSource V2: Handle DELETE commands for group-based sources

2022-02-01 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-38085: Summary: DataSource V2: Handle DELETE commands for group-based sources Key: SPARK-38085 URL: https://issues.apache.org/jira/browse/SPARK-38085 Project: Spark

[jira] [Updated] (SPARK-35801) SPIP: Row-level operations in Data Source V2

2022-02-01 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Affects Version/s: 3.3.0 (was: 3.2.0) > SPIP: Row-level

[jira] [Commented] (SPARK-35801) SPIP: Row-level operations in Data Source V2

2022-02-01 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485502#comment-17485502 ] Anton Okolnychyi commented on SPARK-35801: -- [~viirya], shall we keep this one open until the

[jira] [Updated] (SPARK-35801) SPIP: Row-level operations in Data Source V2

2022-02-01 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Labels: SPIP (was: ) > SPIP: Row-level operations in Data Source V2 >

[jira] [Commented] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2022-01-31 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484985#comment-17484985 ] Anton Okolnychyi commented on SPARK-34183: -- I am sorry it took me a while to come back to this.

[jira] [Commented] (SPARK-34230) Let AQE determine the right parallelism in DistributionAndOrderingUtils

2021-11-09 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441509#comment-17441509 ] Anton Okolnychyi commented on SPARK-34230: -- We should also double check what impact SPARK-36315

[jira] [Commented] (SPARK-34230) Let AQE determine the right parallelism in DistributionAndOrderingUtils

2021-11-05 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17439399#comment-17439399 ] Anton Okolnychyi commented on SPARK-34230: -- A bit more context

[jira] [Commented] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2021-09-12 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413895#comment-17413895 ] Anton Okolnychyi commented on SPARK-34183: -- Sorry for the delay, [~hyukjin.kwon]

[jira] [Commented] (SPARK-33807) Data Source V2: Remove read specific distributions

2021-08-12 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397875#comment-17397875 ] Anton Okolnychyi commented on SPARK-33807: -- I don't think it should be a blocker. We created

[jira] [Commented] (SPARK-33807) Data Source V2: Remove read specific distributions

2021-08-12 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397876#comment-17397876 ] Anton Okolnychyi commented on SPARK-33807: -- Sorry for the delay commenting! > Data Source V2:

[jira] [Created] (SPARK-35900) Consider whether we need to broadcast dynamic filtering subquery results for v2 tables

2021-06-25 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-35900: Summary: Consider whether we need to broadcast dynamic filtering subquery results for v2 tables Key: SPARK-35900 URL: https://issues.apache.org/jira/browse/SPARK-35900

[jira] [Created] (SPARK-35899) Add a utility to convert connector expressions to Catalyst expressions

2021-06-25 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-35899: Summary: Add a utility to convert connector expressions to Catalyst expressions Key: SPARK-35899 URL: https://issues.apache.org/jira/browse/SPARK-35899

[jira] [Updated] (SPARK-35801) SPIP: Row-level operations in Data Source V2

2021-06-24 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Summary: SPIP: Row-level operations in Data Source V2 (was: SPIP: Support MERGE in

[jira] [Updated] (SPARK-35801) SPIP: Support MERGE in Data Source V2

2021-06-24 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Shepherd: L. C. Hsieh Description: Row-level operations such as UPDATE, DELETE,

[jira] [Updated] (SPARK-35801) SPIP: Support MERGE in Data Source V2

2021-06-21 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Description: [MERGE INTO|https://en.wikipedia.org/wiki/Merge_(SQL)] is well suited to

[jira] [Created] (SPARK-35801) SPIP: Support MERGE in Data Source V2

2021-06-17 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-35801: Summary: SPIP: Support MERGE in Data Source V2 Key: SPARK-35801 URL: https://issues.apache.org/jira/browse/SPARK-35801 Project: Spark Issue Type:

[jira] [Created] (SPARK-35779) Support dynamic filtering for v2 tables

2021-06-15 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-35779: Summary: Support dynamic filtering for v2 tables Key: SPARK-35779 URL: https://issues.apache.org/jira/browse/SPARK-35779 Project: Spark Issue Type:

[jira] [Created] (SPARK-34586) Consider supporting distribution and ordering definitions during table creation

2021-03-01 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34586: Summary: Consider supporting distribution and ordering definitions during table creation Key: SPARK-34586 URL: https://issues.apache.org/jira/browse/SPARK-34586

[jira] [Created] (SPARK-34585) Remove BatchWriteHelper completely

2021-03-01 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34585: Summary: Remove BatchWriteHelper completely Key: SPARK-34585 URL: https://issues.apache.org/jira/browse/SPARK-34585 Project: Spark Issue Type:

[jira] [Commented] (SPARK-33736) Handle MERGE in ReplaceNullWithFalseInPredicate

2021-02-17 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286104#comment-17286104 ] Anton Okolnychyi commented on SPARK-33736: -- PR is on the way. > Handle MERGE in

[jira] [Created] (SPARK-34457) DataSource V2: Add default null ordering to SortDirection

2021-02-17 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34457: Summary: DataSource V2: Add default null ordering to SortDirection Key: SPARK-34457 URL: https://issues.apache.org/jira/browse/SPARK-34457 Project: Spark

[jira] [Created] (SPARK-34456) Remove unused write options from BatchWriteHelper

2021-02-17 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34456: Summary: Remove unused write options from BatchWriteHelper Key: SPARK-34456 URL: https://issues.apache.org/jira/browse/SPARK-34456 Project: Spark

[jira] [Created] (SPARK-34230) Let AQE determine the right parallelism in DistributionAndOrderingUtils

2021-01-25 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34230: Summary: Let AQE determine the right parallelism in DistributionAndOrderingUtils Key: SPARK-34230 URL: https://issues.apache.org/jira/browse/SPARK-34230

[jira] [Commented] (SPARK-34184) Remove redundant repartition nodes in the optimizer

2021-01-25 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271632#comment-17271632 ] Anton Okolnychyi commented on SPARK-34184: -- {code:java} ignore("ordered distribution and

[jira] [Created] (SPARK-34184) Remove redundant repartition nodes in the optimizer

2021-01-20 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34184: Summary: Remove redundant repartition nodes in the optimizer Key: SPARK-34184 URL: https://issues.apache.org/jira/browse/SPARK-34184 Project: Spark

[jira] [Updated] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2021-01-20 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-34183: - Priority: Blocker (was: Major) > DataSource V2: Support required distribution and

[jira] [Created] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2021-01-20 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34183: Summary: DataSource V2: Support required distribution and ordering in SS Key: SPARK-34183 URL: https://issues.apache.org/jira/browse/SPARK-34183 Project:

[jira] [Commented] (SPARK-33507) Improve and fix cache behavior in v1 and v2

2021-01-15 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266479#comment-17266479 ] Anton Okolnychyi commented on SPARK-33507: -- [~csun], shall we also handle streaming writes? >

[jira] [Created] (SPARK-34049) DataSource V2: Use Write abstraction in StreamExecution

2021-01-08 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34049: Summary: DataSource V2: Use Write abstraction in StreamExecution Key: SPARK-34049 URL: https://issues.apache.org/jira/browse/SPARK-34049 Project: Spark

[jira] [Commented] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-06 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259592#comment-17259592 ] Anton Okolnychyi commented on SPARK-34026: -- I am going to submit a PR soon. > DataSource V2:

[jira] [Created] (SPARK-34026) DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

2021-01-06 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-34026: Summary: DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering Key: SPARK-34026 URL:

[jira] [Commented] (SPARK-33736) Handle MERGE in ReplaceNullWithFalseInPredicate

2021-01-04 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258222#comment-17258222 ] Anton Okolnychyi commented on SPARK-33736: -- Sorry, I was on holidays. Will get back to the PR

[jira] [Created] (SPARK-33917) Fix misleading error message for unresolved attributes inside INSERT action of MERGE INTO

2020-12-26 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-33917: Summary: Fix misleading error message for unresolved attributes inside INSERT action of MERGE INTO Key: SPARK-33917 URL: https://issues.apache.org/jira/browse/SPARK-33917

[jira] [Created] (SPARK-33868) Normalize expressions passes to DS V2 from OverwriteByExpression

2020-12-21 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-33868: Summary: Normalize expressions passes to DS V2 from OverwriteByExpression Key: SPARK-33868 URL: https://issues.apache.org/jira/browse/SPARK-33868 Project:

[jira] [Updated] (SPARK-33808) DataSource V2: Build logical writes in the optimizer

2020-12-16 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-33808: - Parent: SPARK-22386 Issue Type: Sub-task (was: Bug) > DataSource V2: Build

[jira] [Created] (SPARK-33808) DataSource V2: Build logical writes in the optimizer

2020-12-16 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-33808: Summary: DataSource V2: Build logical writes in the optimizer Key: SPARK-33808 URL: https://issues.apache.org/jira/browse/SPARK-33808 Project: Spark

[jira] [Created] (SPARK-33807) Data Source V2: Remove read specific distributions

2020-12-16 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-33807: Summary: Data Source V2: Remove read specific distributions Key: SPARK-33807 URL: https://issues.apache.org/jira/browse/SPARK-33807 Project: Spark

  1   2   >