[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159462#comment-14159462 ] Lefty Leverenz commented on HIVE-6319: -- NO_AUTO_COMPACTION is documented in the DDL wikidoc now, please review and correct if necessary: * [DDL -- Create Table (see bullet list after the syntax) | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable] > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100231#comment-14100231
]
Lefty Leverenz commented on HIVE-6319:
--
Doc question for [~alangates]: What is the syntax for setting table property
NO_AUTO_COMPACTION?
The Transactions wikidoc just says: "If a table owner does not wish the system
to automatically determine when to compact, then the table property
NO_AUTO_COMPACTION can be set. This will prevent all automatic compactions."
I'd like to include it in the notes about TBLPROPERTIES in the DDL doc, but it
doesn't resemble the other examples which are lowercase and use dots instead of
underscores. For example, ("hbase.table.name"="table_name"),
("immutable"="true"), and ("orc.compress"="NONE"). I couldn't figure it out
from the patch.
* [Hive Transactions -- Table Properties |
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties]
* [DDL -- Create Table (see bullet list after the syntax) |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable]
> Insert, update, delete functionality needs a compactor
> --
>
> Key: HIVE-6319
> URL: https://issues.apache.org/jira/browse/HIVE-6319
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
> HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf
>
>
> In order to keep the number of delta files from spiraling out of control we
> need a compactor to collect these delta files together, and eventually
> rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965651#comment-13965651 ] Alan Gates commented on HIVE-6319: -- Filed HIVE-6886 to track feedback from Owen that hasn't been addressed yet. > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965461#comment-13965461
]
Hive QA commented on HIVE-6319:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639499/HIVE-6319.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5612 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2205/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2205/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639499
> Insert, update, delete functionality needs a compactor
> --
>
> Key: HIVE-6319
> URL: https://issues.apache.org/jira/browse/HIVE-6319
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
> HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf
>
>
> In order to keep the number of delta files from spiraling out of control we
> need a compactor to collect these delta files together, and eventually
> rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964552#comment-13964552 ] Owen O'Malley commented on HIVE-6319: - In the interest of getting 0.13 finalized, I'm +1 except that: * The temp directory for the compactor should have a unique number. Other comments that we can fix later: * AcidUtils.java ** DELTA_DIGITS, BUCKET_DIGITS - you don't need to make it public, just have CompactorTest.addFile not pad the filename. ** BUCKET_DIGIT_PATTERN, LEGACY_BUCKET_DIGIT_PATTERN - i think you can use AcidUtils.parseBaseBucketFilename to get the information you need instead of adding these. ** LEGACY_BUCKET_DIGIT_PATTERN - are you sure that is the right length? * Cleaner.java ** run *** can the unlock throw? If so, we can lose the original exception. I'd suggest replacing the finally with a try/catch with the unlock done in both branches. * CompactorMR.java ** run *** i assume the sd is either the partition sd for partitioned tables or the table sd for non-partitioned ones. you should probably comment that. *** do partition sds have the partition directory as their location? *** move delta check before dealing with the bases *** i would have expected the delta processing to happen in getinputsplits so that you wouldn't need to serialize as much and it wouldn't need to restat the files. ** CompactorInputFormat.getSplits *** we should fix the raw reader so that you can just generate a job per a bucket and buckets that don't exist will just get a 0 row iterator. *** you don't need to build the precise map of deltas for each bucket, the final version of the raw reader will ignore missing files. ** CompactorMap.getWriter *** you should throw an exception if the min or max txn id isn't set. ** CompactorInputSplit *** Could use Arrays.asList to build the List. ** StringableList.toString *** if size() > 0 is redundant > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962543#comment-13962543
]
Hive QA commented on HIVE-6319:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639048/HIVE-6319.patch
{color:green}SUCCESS:{color} +1 5591 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2169/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2169/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639048
> Insert, update, delete functionality needs a compactor
> --
>
> Key: HIVE-6319
> URL: https://issues.apache.org/jira/browse/HIVE-6319
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
> HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf
>
>
> In order to keep the number of delta files from spiraling out of control we
> need a compactor to collect these delta files together, and eventually
> rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961630#comment-13961630 ] Alan Gates commented on HIVE-6319: -- Responses to Ashutosh's comments on review board. Will upload new patch shortly. > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961097#comment-13961097
]
Hive QA commented on HIVE-6319:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638553/HIVE-6319.patch
{color:green}SUCCESS:{color} +1 5589 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2126/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2126/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638553
> Insert, update, delete functionality needs a compactor
> --
>
> Key: HIVE-6319
> URL: https://issues.apache.org/jira/browse/HIVE-6319
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
> HIVE-6319.patch, HiveCompactorDesign.pdf
>
>
> In order to keep the number of delta files from spiraling out of control we
> need a compactor to collect these delta files together, and eventually
> rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960200#comment-13960200 ] Alan Gates commented on HIVE-6319: -- [~ashutoshc] Done. > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960186#comment-13960186 ] Ashutosh Chauhan commented on HIVE-6319: [~alangates] Can you update RB with your latest patch ? > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, > HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958488#comment-13958488
]
Hive QA commented on HIVE-6319:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638222/HIVE-6319.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5584 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2087/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2087/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638222
> Insert, update, delete functionality needs a compactor
> --
>
> Key: HIVE-6319
> URL: https://issues.apache.org/jira/browse/HIVE-6319
> Project: Hive
> Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
> HiveCompactorDesign.pdf
>
>
> In order to keep the number of delta files from spiraling out of control we
> need a compactor to collect these delta files together, and eventually
> rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951697#comment-13951697 ] Alan Gates commented on HIVE-6319: -- Review board posted https://reviews.apache.org/r/19817/ > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949845#comment-13949845 ] Alan Gates commented on HIVE-6319: -- Ran tests locally, all looks good. > Insert, update, delete functionality needs a compactor > -- > > Key: HIVE-6319 > URL: https://issues.apache.org/jira/browse/HIVE-6319 > Project: Hive > Issue Type: Sub-task >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.13.0 > > Attachments: 6319.wip.patch, HIVE-6319.patch, HiveCompactorDesign.pdf > > > In order to keep the number of delta files from spiraling out of control we > need a compactor to collect these delta files together, and eventually > rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
