[
https://issues.apache.org/jira/browse/HIVE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825619#comment-13825619
]
Edward Capriolo commented on HIVE-5317:
---------------------------------------
By the way. I do work like this very often, and having tables that update
periodically cause a lot of problems. The first is when you have to re-compute
a result 4 days later.
You do not want a fresh up-to-date table, you want the table as it existed 4
days ago. When you want to troubleshoot a result you do not want your
intermediate tables trampled over. When you want to rebuild a months worth of
results you want to launch 31 jobs in parallel not 31 jobs in series.
In fact in programming hive I suggest ALWAYS partitioning this dimension tables
by time and NOT doing what this ticket is describing for the reasons above (and
more)
> Implement insert, update, and delete in Hive with full ACID support
> -------------------------------------------------------------------
>
> Key: HIVE-5317
> URL: https://issues.apache.org/jira/browse/HIVE-5317
> Project: Hive
> Issue Type: New Feature
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: InsertUpdatesinHive.pdf
>
>
> Many customers want to be able to insert, update and delete rows from Hive
> tables with full ACID support. The use cases are varied, but the form of the
> queries that should be supported are:
> * INSERT INTO tbl SELECT …
> * INSERT INTO tbl VALUES ...
> * UPDATE tbl SET … WHERE …
> * DELETE FROM tbl WHERE …
> * MERGE INTO tbl USING src ON … WHEN MATCHED THEN ... WHEN NOT MATCHED THEN
> ...
> * SET TRANSACTION LEVEL …
> * BEGIN/END TRANSACTION
> Use Cases
> * Once an hour, a set of inserts and updates (up to 500k rows) for various
> dimension tables (eg. customer, inventory, stores) needs to be processed. The
> dimension tables have primary keys and are typically bucketed and sorted on
> those keys.
> * Once a day a small set (up to 100k rows) of records need to be deleted for
> regulatory compliance.
> * Once an hour a log of transactions is exported from a RDBS and the fact
> tables need to be updated (up to 1m rows) to reflect the new data. The
> transactions are a combination of inserts, updates, and deletes. The table is
> partitioned and bucketed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
