[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432335#comment-15432335 ] Lefty Leverenz commented on HIVE-14035: --- Okay, I'll restore the TODOC2.2 label as a reminder to add the design doc to the wiki. Thanks Eugene. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431610#comment-15431610 ] Eugene Koifman commented on HIVE-14035: --- Sorry, I realized my comment was not clear. I was referring to the hive.txn.operational.properties, not the whole feature. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429915#comment-15429915 ] Lefty Leverenz commented on HIVE-14035: --- Okay, thanks Eugene. Removing the TODOC2.2 label. (You can add it back again if the design doc belongs in the wiki.) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429894#comment-15429894 ] Eugene Koifman commented on HIVE-14035: --- This should not be documented - it's for internal purposes only. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420238#comment-15420238 ] Lefty Leverenz commented on HIVE-14035: --- Here's how you can get wiki edit privileges: * [About This Wiki -- How to get permission to edit | https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit] > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420007#comment-15420007 ] Saket Saurabh commented on HIVE-14035: -- Thanks [~leftylev]. Sure I will add it as a wiki page too. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419834#comment-15419834 ] Lefty Leverenz commented on HIVE-14035: --- Doc note: Besides the design document, which should be added to the wiki, there is a new configuration parameter (*hive.txn.operational.properties*) that will need to be documented in the wiki for release 2.2.0. * [Configuration Properties -- Transactions and Compactor | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-TransactionsandCompactor] Added a TODOC2.2 label. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419832#comment-15419832 ] Lefty Leverenz commented on HIVE-14035: --- [~saketj], I added a link to your design document (version 2) in the wiki's Design Docs. Perhaps the link will become out-of-date if you revise the doc again, but eventually the doc should be reformatted as a wiki page. * [Design Docs -- In Progress | https://cwiki.apache.org/confluence/display/Hive/DesignDocs#DesignDocs-InProgress] > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: Design.Document.Improving ACID performance in > Hive.01.docx, Design.Document.Improving ACID performance in Hive.02.docx, > HIVE-14035.02.patch, HIVE-14035.03.patch, HIVE-14035.04.patch, > HIVE-14035.05.patch, HIVE-14035.06.patch, HIVE-14035.07.patch, > HIVE-14035.08.patch, HIVE-14035.09.patch, HIVE-14035.10.patch, > HIVE-14035.11.patch, HIVE-14035.12.patch, HIVE-14035.13.patch, > HIVE-14035.14.patch, HIVE-14035.15.patch, HIVE-14035.16.patch, > HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419309#comment-15419309 ] Saket Saurabh commented on HIVE-14035: -- Thanks [~ekoifman] and [~sershe] for the review. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch, > HIVE-14035.16.patch, HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419167#comment-15419167 ] Eugene Koifman commented on HIVE-14035: --- committed to master. Thanks [~saketj] for the contribution > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Fix For: 2.2.0 > > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch, > HIVE-14035.16.patch, HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419061#comment-15419061 ] Eugene Koifman commented on HIVE-14035: --- only test failure with age > 1 is transform_ppr1 which also failed in https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/856/testReport/ > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch, > HIVE-14035.16.patch, HIVE-14035.17.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418698#comment-15418698
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12823391/HIVE-14035.17.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10445 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-dynamic_partition_pruning.q-vector_char_mapjoin1.q-unionDistinct_2.q-and-12-more
- did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hive.hcatalog.listener.TestMsgBusConnection.testConnection
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/861/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/861/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-861/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12823391 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch,
> HIVE-14035.16.patch, HIVE-14035.17.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418248#comment-15418248
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12823322/HIVE-14035.16.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10457 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/858/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/858/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-858/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12823322 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch,
> HIVE-14035.16.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417964#comment-15417964 ] Eugene Koifman commented on HIVE-14035: --- +1 patch 16 pending bot run > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch, > HIVE-14035.16.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417754#comment-15417754 ] Sergey Shelukhin commented on HIVE-14035: - +1 pending others' comments > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch, > HIVE-14035.16.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417389#comment-15417389
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12823208/HIVE-14035.15.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10445 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolution
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testACIDwithSchemaEvolution
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/853/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/853/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-853/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12823208 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417183#comment-15417183
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12823203/HIVE-14035.14.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10447 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolution
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testACIDwithSchemaEvolution
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/852/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/852/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-852/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12823203 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.13.patch, HIVE-14035.14.patch, HIVE-14035.15.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414874#comment-15414874 ] Lefty Leverenz commented on HIVE-14035: --- So did I. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414797#comment-15414797 ] Eugene Koifman commented on HIVE-14035: --- Added comments on RB > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414383#comment-15414383 ] Saket Saurabh commented on HIVE-14035: -- [~sershe] Thanks for the comments on RB. I am working on fixing those. No, the last run for patch 13 did not have split-update enabled by default. There are many tests that assert on number of files and directory layout that would anyway fail in PTest if we run those tests w/o modification. However, excluding those assert failures, when I ran these locally, the only other failures were NegativeArrayIndexException & IndexOutOfBoundException caused by HIVE-14448 and not related to this patch. However, I have created a subclass TestTxnCommands3 that should ideally mimic this behavior with split-update enabled by default for a large number of ACID scenarios. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414374#comment-15414374 ] Eugene Koifman commented on HIVE-14035: --- Can't make this the default yet because it will break RU (the downgrade part specifically). > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414362#comment-15414362 ] Sergey Shelukhin commented on HIVE-14035: - Few minor comments on RB. Was the last HiveQA run with split-update enabled? Actually, should we have it on by default? cc [~gopalv] Probably still needs review from someone familiar with ACID. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.13.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413127#comment-15413127
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12822704/HIVE-14035.13.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10466 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-script_pipe.q-orc_ppd_schema_evol_2a.q-join1.q-and-12-more
- did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
TestQueryLifeTimeHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/826/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/826/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-826/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12822704 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.13.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408659#comment-15408659 ] Saket Saurabh commented on HIVE-14035: -- Thanks [~sershe] for the comments and the feedback. I am working on addressing them. Sure, I will also put up a separate patch with default enabled for HiveQA run. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408584#comment-15408584 ] Sergey Shelukhin commented on HIVE-14035: - Actually, one good thing to do would be to enable it by default in the next patch and see what HiveQA run, with all the existing ACID tests running in new mode, shows. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408536#comment-15408536 ] Sergey Shelukhin commented on HIVE-14035: - Also, I think q file tests are needed... > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408527#comment-15408527 ] Sergey Shelukhin commented on HIVE-14035: - Some CR feedback on RB, mostly minor. Someone more familiar with ACID should review the core logic, ideally. It makes sense to me. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396742#comment-15396742
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820561/HIVE-14035.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10387 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/664/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/664/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820561 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch,
> HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366408#comment-15366408 ] Saket Saurabh commented on HIVE-14035: -- Link for review board for this patch: https://reviews.apache.org/r/49766/ Thanks > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364781#comment-15364781 ] Eugene Koifman commented on HIVE-14035: --- Note to self: PPD check can be based on file name - no need read footer to get Acid stats > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363805#comment-15363805
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816335/HIVE-14035.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10299 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/377/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/377/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-377/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816335 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360349#comment-15360349
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815890/HIVE-14035.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10292 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/354/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/354/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-354/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815890 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.07.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359954#comment-15359954
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815830/HIVE-14035.06.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10283 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucket6.q-scriptfile1_win.q-quotedid_smb.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/342/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/342/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-342/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815830 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353912#comment-15353912
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814052/HIVE-14035.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 10284 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_acid3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join_acid_non_acid
org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDump
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptDataFiles
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptSideFiles
org.apache.hive.hcatalog.streaming.TestStreaming.testInterleavedTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testMultipleTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344531#comment-15344531
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812327/HIVE-14035.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10253 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/220/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/220/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812327 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339687#comment-15339687
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811785/HIVE-14035.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10235 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/190/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/190/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-190/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12811785 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336439#comment-15336439
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811174/HIVE-14035.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 10234 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.TestTxnCommands.testDelete
org.apache.hadoop.hive.ql.TestTxnCommands.testDeleteIn
org.apache.hadoop.hive.ql.TestTxnCommands.testMultipleDelete
org.apache.hadoop.hive.ql.TestTxnCommands.testUpdateDeleteOfInserts
org.apache.hadoop.hive.ql.TestTxnCommands2.testAlterTable
org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testNewBaseAndDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderNewBaseAndDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderOldBaseAndDelta
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.schemaEvolutionAddColDynamicPartitioningUpdate
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testUpdatesAndDeletes
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/151/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/151/console
Test logs:
