[
https://issues.apache.org/jira/browse/SQOOP-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333491#comment-16333491
]
Daniel Voros commented on SQOOP-3267:
-------------------------------------
[~vasas], [~maugli] thank you both for your replies!
I agree with you on keeping the history. I see two ways to do that.
Option A) is a sort-of workaround. Let users know that Sqoop will delete all
previous versions of columns when updating them to NULL in the source table and
ask them to set KEEP_DELETED_CELLS on the underlying HBase table if they still
want to preserve history.
Option B) is inserting an empty string for NULL values. In detail:
- Whenever we're importing a NULL value (whether if it's from a new row or an
updated one) we insert a special string (let's call it NULL_STRING). Note, that
it would be better to insert NULL, but HBase lacks the notion of NULL.
- The value of NULL_STRING is "" (empty string) by default but is
configurable. (Probably via the already existing {{--null-string}} argument.)
- This behavior should NOT depend on the incremental mode ("append" or
"lastmodified").
Two notes:
- Option B) is similar to what was introduced in Phoenix in PHOENIX-1578.
(When using the STORE_NULLS=true table option, there's no way to tell a NULL
value from an empty string in Phoenix.)
- I've also tested how Hive treats NULL values when storing a table in HBase.
Empty (or deleted) columns are displayed as NULL when reading the table, but
updating a column to NULL in Hive fails (see HIVE-3336).
> Incremental import to HBase deletes only last version of column
> ---------------------------------------------------------------
>
> Key: SQOOP-3267
> URL: https://issues.apache.org/jira/browse/SQOOP-3267
> Project: Sqoop
> Issue Type: Bug
> Components: hbase-integration
> Affects Versions: 1.4.7
> Reporter: Daniel Voros
> Assignee: Daniel Voros
> Priority: Major
> Attachments: SQOOP-3267.1.patch
>
>
> Deletes are supported since SQOOP-3149, but we're only deleting the last
> version of a column when the corresponding cell was set to NULL in the source
> table.
> This can lead to unexpected and misleading results if the row has been
> transferred multiple times, which can easily happen if it's being modified on
> the source side.
> Also SQOOP-3149 is using a new Put command for every column instead of a
> single Put per row as before. This could probably lead to a performance drop
> for wide tables (for which HBase is otherwise usually recommended).
> [~jilani], [~anna.szonyi] could you please comment on what you think would be
> the expected behavior here?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
