[ https://issues.apache.org/jira/browse/SQOOP-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333491#comment-16333491 ]
Daniel Voros commented on SQOOP-3267: ------------------------------------- [~vasas], [~maugli] thank you both for your replies! I agree with you on keeping the history. I see two ways to do that. Option A) is a sort-of workaround. Let users know that Sqoop will delete all previous versions of columns when updating them to NULL in the source table and ask them to set KEEP_DELETED_CELLS on the underlying HBase table if they still want to preserve history. Option B) is inserting an empty string for NULL values. In detail: - Whenever we're importing a NULL value (whether if it's from a new row or an updated one) we insert a special string (let's call it NULL_STRING). Note, that it would be better to insert NULL, but HBase lacks the notion of NULL. - The value of NULL_STRING is "" (empty string) by default but is configurable. (Probably via the already existing {{--null-string}} argument.) - This behavior should NOT depend on the incremental mode ("append" or "lastmodified"). Two notes: - Option B) is similar to what was introduced in Phoenix in PHOENIX-1578. (When using the STORE_NULLS=true table option, there's no way to tell a NULL value from an empty string in Phoenix.) - I've also tested how Hive treats NULL values when storing a table in HBase. Empty (or deleted) columns are displayed as NULL when reading the table, but updating a column to NULL in Hive fails (see HIVE-3336). > Incremental import to HBase deletes only last version of column > --------------------------------------------------------------- > > Key: SQOOP-3267 > URL: https://issues.apache.org/jira/browse/SQOOP-3267 > Project: Sqoop > Issue Type: Bug > Components: hbase-integration > Affects Versions: 1.4.7 > Reporter: Daniel Voros > Assignee: Daniel Voros > Priority: Major > Attachments: SQOOP-3267.1.patch > > > Deletes are supported since SQOOP-3149, but we're only deleting the last > version of a column when the corresponding cell was set to NULL in the source > table. > This can lead to unexpected and misleading results if the row has been > transferred multiple times, which can easily happen if it's being modified on > the source side. > Also SQOOP-3149 is using a new Put command for every column instead of a > single Put per row as before. This could probably lead to a performance drop > for wide tables (for which HBase is otherwise usually recommended). > [~jilani], [~anna.szonyi] could you please comment on what you think would be > the expected behavior here? -- This message was sent by Atlassian JIRA (v7.6.3#76005)