[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131437#comment-15131437 ] Eugene Koifman commented on HIVE-11716: --- branch-1: caf4b516b74201633e25d99bfcf5d69eacc8a617 > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 2.1.0 > > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, > HIVE-11716.6.patch, HIVE-11716.branch-1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130736#comment-15130736
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785860/HIVE-11716.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10034 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-vector_null_projection.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6852/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6852/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6852/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785860 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130958#comment-15130958 ] Eugene Koifman commented on HIVE-11716: --- [~wzheng] could you make a branch-1 patch > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130854#comment-15130854 ] Wei Zheng commented on HIVE-11716: -- TestHiveMetaStorePartitionSpecs passed locally for me without any failures. My change shouldn't have impact on that test. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131030#comment-15131030 ] Eugene Koifman commented on HIVE-11716: --- committed to master https://github.com/apache/hive/commit/f2e9edbc2d81e32f5da700cd7024959184aeaebf > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15129337#comment-15129337 ] Eugene Koifman commented on HIVE-11716: --- +1 pending tests > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127452#comment-15127452 ] Eugene Koifman commented on HIVE-11716: --- ErrorMsg.TXNMGR_NOT_ACID - this should be a parametrized message. At minimum, it should include db_name.table_name. Perhaps it should also include first 10 or 20 chars of the command itself to make easier to read the logs down the road. I don't think SessionState.initTxnMgr() works long term. What if we add BetterTxnManger later? I would, in addition to txnMgr also add txnMgrName to remember the last property name or perhaps better add HiveTxnManager.getTxnManagerName() so that you can query currently installed manager for its name. Nit: In TestDbTxnManager2.testDummyTxnManagerOnAcidTable(). Instead of 10265, etc could you use ErrorMsg.TXNMGR_NOT_ACID, etc > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127702#comment-15127702 ] Wei Zheng commented on HIVE-11716: -- Good advice. Will follow up with new patch. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126748#comment-15126748 ] Wei Zheng commented on HIVE-11716: -- Mismatches are unrelated > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125238#comment-15125238
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785122/HIVE-11716.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10032 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6809/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785122 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123118#comment-15123118
]
Wei Zheng commented on HIVE-11716:
--
[~alangates] You're right. The mismatch is caused by my change, but in a good
way :)
Basically the test result mismatch is like this:
{code}
< FAILED: SemanticException [Error 10297]: Attempt to do update or delete on
table default.acid_notbucketed that does not use an AcidOutputFormat or is not
bucketed
---
> FAILED: SemanticException [Error 10297]: Attempt to do update or delete on
> table acid_notbucketed that does not use an AcidOutputFormat or is not
> bucketed
{code}
Without patch the error was thrown by SemanticAnalyzer#checkAcidConstraints,
there we have
{code}
if (table.getNumBuckets() < 1) {
throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE,
table.getTableName());
}
{code}
where table.getTableName() just returns tablename acid_notbucketed.
With patch the error was caught earlier and thrown by
SemanticAnalyzer#getMetaData()
{code}
// Disallow update and delete on non-acid tables
if ((updating() || deleting()) && !isAcid && isTableWrittenTo) {
//isTableWrittenTo: delete from acidTbl where a in (select id from
nonAcidTable)
//so only assert this if we are actually writing to this table
// isAcidTable above also checks for whether we are using an acid
compliant
// transaction manager. But that has already been caught in
// UpdateDeleteSemanticAnalyzer, so if we are updating or deleting
and getting nonAcid
// here, it means the table itself doesn't support it.
throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE,
tab_name);
}
{code}
where tab_name is default.acid_notbucketed
Here isAcid flag was true without patch, which is wrong, because isAcidTable()
has a bug as it only checked 'transactional'='true', but didn't check whether
the table is bucketed or whether it's ORC. The patch fixed isAcidTable(), so
isAcid is false now, thus we can trigger this logic.
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch, HIVE-11716.4.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122616#comment-15122616 ] Wei Zheng commented on HIVE-11716: -- Both delete_not_bucketed.q and update_not_bucketed.q passed locally for me. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122676#comment-15122676 ] Alan Gates commented on HIVE-11716: --- Those tests don't generally fail and given that you're working exactly in that space I wouldn't ignore those failures. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122531#comment-15122531
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784780/HIVE-11716.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10022 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_delete_not_bucketed
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_update_not_bucketed
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6782/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6782/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6782/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784780 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch, HIVE-11716.4.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120102#comment-15120102
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784545/HIVE-11716.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10035 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6762/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6762/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6762/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784545 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117995#comment-15117995
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784367/HIVE-11716.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6749/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6749/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6749/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6749/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
eb4a164..27172bc master -> origin/master
+ git reset --hard HEAD
HEAD is now at eb4a164 HIVE-12905 : Issue with mapjoin in tez under certain
conditions (Vikram Dixit K, reviewed by Sergey Shelukhin)
+ git clean -f -d
Removing
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetChangeVersionRequest.java
Removing
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetChangeVersionResult.java
Removing
metastore/src/model/org/apache/hadoop/hive/metastore/model/MChangeVersion.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 27172bc HIVE-12889: Support COUNT(DISTINCT) for partitioning
query. (Aihua Xu, reviewed by Szehon Ho)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784367 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118091#comment-15118091 ] Eugene Koifman commented on HIVE-11716: --- I think you need to use AcidUtil.isTablePropertyTransactional() to see if table is transactional rather than AcidUtils.isAcidTable(tab). The later will return false for table that was non-acid-to-acid upgraded but w/o major compaction having been run. Maybe even better to define a new isAcidTable(Table t) method which checks that it's bucketed, implements AcidInput/AcidOutputFormat and has the transactional=true. Also, SemanticAnalyzer.isAcidTable() should delegate to new method (or be removed) or at minimum not check the which TxnManager is installed > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118097#comment-15118097 ] Eugene Koifman commented on HIVE-11716: --- one other comment: now that you've implemented logic to raise an error when Acid table is accessed from non-acid session, it's seems like it's safe to allow user to change TxnManager within a Session. Since it can no longer do any harm. So maybe using HIV_IN_TEST isn't needed... > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118134#comment-15118134 ] Eugene Koifman commented on HIVE-11716: --- Nit: "Cannot query an ACID table using a non-ACID transaction manager" - maybe should read "This command is not allowed on an ACID " > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118150#comment-15118150 ] Wei Zheng commented on HIVE-11716: -- How about "This command is not allowed on an ACID table with a non-ACID transaction manager"? > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118160#comment-15118160 ] Wei Zheng commented on HIVE-11716: -- Sounds good. Now users can change the TxnManager for real. But one issue is, initialize a TxnManager for every query might be a little slower than reusing the same one in a session (because we will check conf every time and init a TxnManager) > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118165#comment-15118165 ] Wei Zheng commented on HIVE-11716: -- Unless I compare the original conf and the passed-in conf, and only initialize a new TxnManger if hive.txn.manager field is changed? > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118166#comment-15118166 ] Eugene Koifman commented on HIVE-11716: --- I think it should only initialized if it is not already set or if there is explicit "set hive.txn.manager..." command. In the later case, the previous instance must be shutdown first. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118183#comment-15118183 ] Wei Zheng commented on HIVE-11716: -- But I think set hive.txn.manager= ... will make changes to conf, and we will only know this if we do a comparison between confs > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118194#comment-15118194 ] Eugene Koifman commented on HIVE-11716: --- in that case you are right, you should keep track of hive.txn.manager field > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116835#comment-15116835 ] Wei Zheng commented on HIVE-11716: -- This JIRA solves the problems below: * Explicitly disallow switching hive.txn.manager (This was not allowed always, but we didn't throw out error, and SET command gives people impression that this entry can be changed because it does show the new value, which is misleading) * For queries against ACID table with a transaction manager that doesn't support ACID, we raise errors. This applies to SELECT, INSERT INTO, INSERT OVERWRITE, UPDATE and DELETE. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116837#comment-15116837 ] Wei Zheng commented on HIVE-11716: -- I will follow up with HIVE-12441 once this one is done > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113039#comment-15113039 ] Eugene Koifman commented on HIVE-11716: --- HIVE-12441 is related since in both cases we need to walk the plan and see if there are any ACID tables in it (read or written) > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
