[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947620#comment-13947620 ] Lefty Leverenz commented on HIVE-5843: -- This introduces 13 new config params, but HIVE-6606 removes two of them. * hive.txn.manager, -hive.txn.driver- (removed), -hive.txn.connection.string- (removed), hive.txn.timeout, hive.txn.max.open.batch, hive.txn.testing, hive.compactor.initiator.on, hive.compactor.worker.threads, hive.compactor.worker.timeout, hive.compactor.check.interval, hive.compactor.delta.num.threshold, hive.compactor.delta.pct.threshold, hive.compactor.abortedtxn.threshold They are listed in a comment on HIVE-6586, so they can be added to the new version of HiveConf.java after HIVE-6037 gets committed. They also need to be documented in the wiki. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.10.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch, > HIVE-5843.4.patch, HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.8.patch, > HIVE-5843.8.src-only.patch, HIVE-5843.9.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919706#comment-13919706 ] Alan Gates commented on HIVE-5843: -- I don't believe this test failure is related to my changes. I ran this test with and without my patch on Mac and Unix and it passed in all scenarios. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.10.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch, > HIVE-5843.4.patch, HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.8.patch, > HIVE-5843.8.src-only.patch, HIVE-5843.9.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[
https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919587#comment-13919587
]
Hive QA commented on HIVE-5843:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632435/HIVE-5843.10.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5348 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1614/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1614/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12632435
> Transaction manager for Hive
>
>
> Key: HIVE-5843
> URL: https://issues.apache.org/jira/browse/HIVE-5843
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.12.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch,
> HIVE-5843-src-only.patch, HIVE-5843.10.patch, HIVE-5843.2.patch,
> HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch,
> HIVE-5843.4.patch, HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.8.patch,
> HIVE-5843.8.src-only.patch, HIVE-5843.9.patch, HIVE-5843.patch,
> HiveTransactionManagerDetailedDesign (1).pdf
>
>
> As part of the ACID work proposed in HIVE-5317 a transaction manager is
> required.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919578#comment-13919578 ] Ashutosh Chauhan commented on HIVE-5843: +1 > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.10.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch, > HIVE-5843.4.patch, HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.8.patch, > HIVE-5843.8.src-only.patch, HIVE-5843.9.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[
https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916214#comment-13916214
]
Hive QA commented on HIVE-5843:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631617/HIVE-5843.9.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5290 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1555/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1555/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12631617
> Transaction manager for Hive
>
>
> Key: HIVE-5843
> URL: https://issues.apache.org/jira/browse/HIVE-5843
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.12.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch,
> HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path,
> HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch,
> HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.8.patch,
> HIVE-5843.8.src-only.patch, HIVE-5843.9.patch, HIVE-5843.patch,
> HiveTransactionManagerDetailedDesign (1).pdf
>
>
> As part of the ACID work proposed in HIVE-5317 a transaction manager is
> required.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913705#comment-13913705 ] Alan Gates commented on HIVE-5843: -- Review board review https://reviews.apache.org/r/18536/ > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path, > HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch, > HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913189#comment-13913189 ] Ashutosh Chauhan commented on HIVE-5843: [~alangates] If you can create RB entry sans generated code to ease review for it, that will be great. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path, > HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch, > HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912361#comment-13912361 ] Lefty Leverenz commented on HIVE-5843: -- Thanks Alan, a doc JIRA seems like a good idea for this. About the nit, I'm sure that "partitions need compacted" sounds wrong -- maybe you meant "need compaction" or maybe I misunderstood the concept. I read it as similar to "The dishes in the sink need cleaned" vs. "need to be cleaned" or "need cleaning." Not so? But I can't cite a grammar rule without doing some research. So far I've found out that "need" can be a regular verb or a modal. "I need to look it up, but I need not obsess over it," respectively. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path, > HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch, > HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912059#comment-13912059 ] Alan Gates commented on HIVE-5843: -- I'm definitely hoping this makes it into 0.13. And no, it isn't only incomplete feature. If it was, I'd wait until after 0.13 branched. HIVE-5687 depends on this, and the hope is to get it into 0.13. As for the comments in HiveConf, I didn't realize I was writing documentation there or I would have paid closer attention to my grammar. However, a nit on your nit. "need compacted." --> "need to be compacted." Are you sure? What is the grammar rule there? Overall on documentation though, there will be a fair amount to write, especially once we have HIVE-6319 and HIVE-6060 there. Should I file a separate JIRA outline the documentation needs? > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path, > HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch, > HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912036#comment-13912036 ] Lefty Leverenz commented on HIVE-5843: -- HiveConf comment nits: * hive.compactor.check.interval: "// Time in seconds between checks to see if any partitions need compacted." --> "need to be compacted." * hive.txn.timeout: "// time after which ..." --> init cap "Time" Also a question: If this goes into Hive 0.13.0, will it be useful immediately or just a piece of an incomplete feature? Thirteen new config parameters are added, and I'm wondering about documentation (as always). When HIVE-6037 gets committed we won't need to update hive-default.xml.template anymore but the parameter comments will have to be moved into the definitions. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 5843.5-wip.patch, HIVE-5843-src-only.6.patch, > HIVE-5843-src-only.patch, HIVE-5843.2.patch, HIVE-5843.3-src.path, > HIVE-5843.3.patch, HIVE-5843.4-src.patch, HIVE-5843.4.patch, > HIVE-5843.6.patch, HIVE-5843.7.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13884782#comment-13884782 ] Alan Gates commented on HIVE-5843: -- The first four tests auto_sortmerge_join_16, import_exported_table, infer_bucket_sort_reducers_power_two, and load_hdfs_file_with_space_in_the_name pass in my local environment. The last one, file_with_header_footer_negative fails for me on both my branch and trunk. Thus I conclude none of these have anything to do with my changes. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch, > HIVE-5843.4.patch, HIVE-5843.patch, HiveTransactionManagerDetailedDesign > (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[
https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881960#comment-13881960
]
Hive QA commented on HIVE-5843:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12624960/HIVE-5843.4.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5052 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1017/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1017/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12624960
> Transaction manager for Hive
>
>
> Key: HIVE-5843
> URL: https://issues.apache.org/jira/browse/HIVE-5843
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.12.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch,
> HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch,
> HIVE-5843.4.patch, HIVE-5843.patch, HiveTransactionManagerDetailedDesign
> (1).pdf
>
>
> As part of the ACID work proposed in HIVE-5317 a transaction manager is
> required.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881201#comment-13881201 ] Brock Noland commented on HIVE-5843: Great to hear! > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.4-src.patch, > HIVE-5843.4.patch, HIVE-5843.patch, HiveTransactionManagerDetailedDesign > (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878974#comment-13878974 ] Brock Noland commented on HIVE-5843: bq. I'll do some micro-benchmarking and if I can't show that it costs something extra I'll move the single integer values to a struct. Awesome! bq. Can thrift handle adding new parameters to functions? It would seem a natural thing to support given thrift's version independence, but I couldn't find anything one way or another in the docs. Agreed, you would think that it would be a priority...but AFAIK that is an issue. Thus the use of a single "Request" object per method call. bq. What's wrong with exceptions in thrift? I suppose the method signatures throw TException which is the parent class of all generated thrift exceptions? If so, then I don't suppose there is an issue. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878842#comment-13878842 ] Alan Gates commented on HIVE-5843: -- bq. Therefore do we see these methods being called in tight loops? It seems that the IO in this situation would dominate any unwrapping of objects? I don't see this in tight loops, but whenever this is turned on we are adding a round trip to the metastore on every read query (for lock acquisition) and two for inserts (txn open plus lock acquisition). I'd just like to keep those as fast as possible. I'll do some micro-benchmarking and if I can't show that it costs something extra I'll move the single integer values to a struct. A question though. Can thrift handle adding new parameters to functions? It would seem a natural thing to support given thrift's version independence, but I couldn't find anything one way or another in the docs. bq. Additionally I would not throw an exceptions as they are problematic as well What's wrong with exceptions in thrift? > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874898#comment-13874898 ] Brock Noland commented on HIVE-5843: bq. In the places where they were taking more than one argument or returning values I've put in structs. Thank you very much!! bq. But a number of the calls take a single long value, the transaction id. I don't want to wrap and unwrap this all the time, as I'd like this to be as fast as possible and it seems overkill. Sorry Alan, please excuse me for my ignorance. From HIVE-5317 "Agreed, this will fail badly in a one insert at a time situation.". Therefore do we see these methods being called in tight loops? It seems that the IO in this situation would dominate any unwrapping of objects? bq. The odds that methods like abortTransaction() will take more than a transactionId seem low. Please know that I am not trying to be obstinate here! :) My only concern is that we follow best practices for new thrift APIs. The number of times I have heard "we won't need to change X" and then X later needed to be changed is too numerous to count. API's always evolve and therefore I would suggest that every method take a single parameter and return a single response. Additionally I would not throw an exceptions as they are problematic as well and instead return a Status struct (code, message, stacktrace) which can contain an optional stack trace should an exception be thrown on the server side. > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch, > HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[
https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874594#comment-13874594
]
Hive QA commented on HIVE-5843:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12623577/HIVE-5843.3.patch
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 5034 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_scan_params
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/946/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/946/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12623577
> Transaction manager for Hive
>
>
> Key: HIVE-5843
> URL: https://issues.apache.org/jira/browse/HIVE-5843
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.12.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-5843-src-only.patch, HIVE-5843.2.patch,
> HIVE-5843.3-src.path, HIVE-5843.3.patch, HIVE-5843.patch,
> HiveTransactionManagerDetailedDesign (1).pdf
>
>
> As part of the ACID work proposed in HIVE-5317 a transaction manager is
> required.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[ https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854941#comment-13854941 ] Brock Noland commented on HIVE-5843: Alan, Can you generate a patch which does not contain the generated code for reviewing purposes? Also, you thrift methods should take a single request object and return a single response object so the methods can be versioned. HS2 does a good job of this: https://github.com/apache/hive/blob/trunk/service/if/TCLIService.thrift#L1020 > Transaction manager for Hive > > > Key: HIVE-5843 > URL: https://issues.apache.org/jira/browse/HIVE-5843 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.12.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: HIVE-5843.2.patch, HIVE-5843.patch, > HiveTransactionManagerDetailedDesign (1).pdf > > > As part of the ACID work proposed in HIVE-5317 a transaction manager is > required. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HIVE-5843) Transaction manager for Hive
[
https://issues.apache.org/jira/browse/HIVE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854873#comment-13854873
]
Hive QA commented on HIVE-5843:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619930/HIVE-5843.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 4820 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_with_concurrency
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_try_drop_locked_db
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/725/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/725/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12619930
> Transaction manager for Hive
>
>
> Key: HIVE-5843
> URL: https://issues.apache.org/jira/browse/HIVE-5843
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.12.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-5843.2.patch, HIVE-5843.patch,
> HiveTransactionManagerDetailedDesign (1).pdf
>
>
> As part of the ACID work proposed in HIVE-5317 a transaction manager is
> required.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
